768204

research-article2018
PBIXXX10.1177/1098300718768204Journal of Positive Behavior InterventionsGage et al.

Empirical Research
Journal of Positive Behavior Interventions

The Impact of School-Wide Positive

1­–10
© Hammill Institute on Disabilities 2018
Reprints and permissions:
Behavior Interventions and Supports on sagepub.com/journalsPermissions.nav
DOI: 10.1177/1098300718768204
https://doi.org/10.1177/1098300718768204

School Suspensions: A Statewide jpbi.sagepub.com

Quasi-Experimental Analysis

Nicholas A. Gage, PhD1, Ahhyun Lee, MS1,

Nicolette Grasley-Boy, MEd, BCBA1,
and Heather Peshak George, PhD2

Abstract
This study explored the effects of School-Wide Positive Behavior Interventions and Supports (SWPBIS) on school
suspensions and behavioral incidents for elementary and intermediate schools in Georgia implementing with fidelity by
comparing results with a propensity score–matched comparison group of schools that never received SWPBIS training.
Significant decreases in suspensions and disciplinary exclusions were found in schools implementing with fidelity compared
with matched comparison schools. Schools implementing SWPBIS with higher fidelity had fewer out-of-school suspensions
and disciplinary incidents than schools implementing with lower levels of fidelity, but both groups had significantly fewer
suspensions and incidents than the comparison group. When converted to standard mean difference effect sizes, results
indicated medium to large effects. These findings suggest that SWPBIS is an effective model for reducing disciplinary
exclusions and disciplinary incidents and that implementing SWPBIS with fidelity can result in meaningful improvements on
student behavioral outcomes in schools.

Keywords
positive behavior support, school-wide intervention(s), disciplinary exclusion

According to the most recent data from the U.S. Department
of Education’s Office of Civil Rights, over three million stu-
dents receive one or more in-school suspension and just over
three million students receive one or more out-of-school sus-
pension each year (U.S. Department of Education, 2016). Of
those being suspended, a student with a disability is over
twice as likely to be suspended as a student without a dis-
ability, while Black males are three and a half times more
likely to be suspended than White students. Although school
suspension, a widely used form of disciplinary exclusion, is
a consequence for a behavioral incident in school (e.g.,
fighting, disrespect, theft), suspensions have concerning
short- and long-term consequences. Suspensions reduce the
amount of time students are in class and receiving instruc-
tion (Losen, Hodson, Keith, Morrison, & Belway, 2015),
decrease student engagement in school (Kupchik & Catlaw,
2015; Mowen & Manierre, 2015) and academic achieve-
ment (Arcia, 2006; Noltemeyer, Ward, & McLoughlin,
2015), and significantly increase the odds of dropping out of
school (Noltemeyer et al., 2015) and being arrested in early
adulthood (Mowen & Brent, 2016).
A recent policy statement from the American Academy of
Pediatrics Council on School Health (2013) highlighted the
deleterious effects of suspensions on students and recom-
mended three strategies to reduce their use in schools: (a)
early intervention, (b) early identification, and (c) the imple-
mentation of School-Wide Positive Behavior Interventions
and Supports (SWPBIS). Of those, SWPBIS is the only rec-
ommendation that has an established framework with clearly
outlined critical components. Yet, although evidence sug-
gests that SWPBIS is an effective framework for reducing
problem behavior and, subsequently, suspensions (Horner,
Sugai, & Anderson, 2010), most of the empirical evidence is
based on case studies and correlational analyses. In fact,
Gage, Whitford, and Katsiyannis (in press) conducted a sys-
tematic review and meta-analysis of experimental (i.e., ran-
domized controlled trials [RCTs]) and quasi-experimental
studies evaluating the impact of SWPBIS on school-level
1
University of Florida, Gainesville, USA
2
University of South Florida, Tampa, USA
Corresponding Author:
Nicholas A. Gage, University of Florida, 1403 Normal Hall, P.O. Box
117050, Gainesville, FL 32611, USA.
Email: gagenicholas@gmail.com
Action Editor: Melissa Stormont
2 Journal of Positive Behavior Interventions 00(0)
rates of disciplinary exclusions, including in- and out-of-
school suspensions and office discipline referrals (ODR).
Overall, they found four experimental studies conducted
with 90 schools total. No treatment effects were found for
ODR, but there was a significant and large effect for school
suspensions (g = –.86). Two of the four studies included sus-
pensions with an effect size based only on a total sample of
just 72 schools. Council for Exceptional Children (CEC)
quality indicators (Gersten et al., 2005) suggested that a
practice be considered promising if (a) at least two high-
quality studies support the practice and (b) the weighted
effect size is greater than 0. Both studies with suspension
outcomes (Algozzine et al., 2012; Flannery, Fenning, Kato,
& McIntosh, 2014) met at least 93% of the CEC quality indi-
cators. Therefore, based on Gersten et al.’s criteria, SWPBIS
is a promising practice, but more research is needed for
determining whether SWPBIS is an evidence-based prac-
tice. Therefore, we conducted a large quasi-experimental
design (QED) study on the treatment effects of SWPBIS on
school-level disciplinary exclusions.
SWPBIS
SWPBIS is a systematic approach for implementing school-
wide evidence-based prevention and intervention practices
using a multitiered system to provide behavioral support for
all students (Horner, Sugai, & Anderson, 2010). SWPBIS is
not a curriculum, intervention, or program (Sugai & Horner,
2009), but a process of building a school’s capacity to (a)
implement effective and preventive behavioral practices
with integrity, (b) make data-based and team-based deci-
sions, and (c) build a positive school climate and culture
leading to school improvement and success (Horner, Sugai,
& Anderson, 2010). SWPBIS is comprised of a continuum
of three tiers of prevention and intervention, including pri-
mary prevention implemented across schools for all stu-
dents, secondary prevention implemented for small groups
of students, and tertiary intervention provided for individu-
als to meet the unique needs of each student. When data
indicate that students are not responsive to the primary pre-
vention efforts (e.g., positively stated behavioral expecta-
tions that are taught and reinforced school-wide, a
continuum for responding to inappropriate behaviors), they
receive targeted interventions, which may include evidence-
based mentoring programs such as Check-In Check-Out
(Crone, Hawken, & Horner, 2010) or small group social
skills lessons (Mitchell, Stormont, & Gage, 2011). Students
who continue to exhibit elevated levels of behavior prob-
lems following targeted intervention can be referred for
intensive or tertiary intervention, which typically involves
development of a functional behavior assessment and a sub-
sequent individualized behavior intervention plan.
SWPBIS has been implemented for over 30 years in more
than 25,000 schools nationally and in numerous countries
(www.pbis.org). Research studies have documented the
impact of SWPBIS on both student- and school-level out-
comes. Empirical evidence documents improvements in dis-
ciplinary consequences, school climate, organizational
health, bullying and peer victimization, and academic
achievement when implemented with fidelity (Bradshaw,
Koth, Thornton, & Leaf, 2009; Bradshaw, Mitchell, & Leaf,
2010; Childs, Kincaid, George, & Gage, 2016; Gage, Leite,
Childs, & Kincaid, 2017; Horner, Sugai, Smolkowski, et al.,
2010; Sadler & Sugai, 2009; Simonsen et al., 2012;
Waasdorp, Bradshaw, & Leaf, 2012). SWPBIS provides
schools with a framework to construct an efficient and effec-
tive prevention continuum of behavioral supports, ensuring
evidence-based practices are accessible to all students across
all settings (Sugai, Simonsen, Bradshaw, Horner, & Lewis,
2014). Therefore, studies evaluating the school-level effects
of SWPBIS are aligned with questions of efficacy and
effectiveness.
Empirical Evidence of School-Level
Effects
A handful of studies have examined the effects of SWPBIS
on student behavioral outcomes, including suspensions, by
comparing schools implementing with fidelity to those not
implementing with fidelity. For example, Simonsen et al.
(2012) examined SWPBIS implementation in 428 Illinois
schools across 7 years and found that schools implementing
SWPBIS with fidelity had significantly fewer school sus-
pensions. However, no differences were found for ODR by
implementation levels. Childs et al. (2016) examined the
relationship between fidelity of implementation and school
suspensions in 1,122 Florida schools across 4 years. Their
findings suggest schools implementing with higher fidelity
had significantly fewer suspensions and ODR. Results from
these studies, along with Freeman et al. (2016), highlighted
the relationship between increased fidelity of implementa-
tion and reduced ODR. However, these studies only com-
pared schools that received SWPBIS training rather than
comparing results to control schools that were not trained.
Two RCT studies and two QED studies examined the
school-level effects of SWPBIS on suspensions and ODR
while including a business-as-usual control group.
Bradshaw et al. (2010) examined the effects of SWPBIS via
a 5-year longitudinal RCT conducted in 37 elementary
schools in Maryland. Overall, the authors found significant
reductions in suspensions in both treatment groups, but
there was no significant difference between treatment and
control groups (d = −0.07). The authors only collected ODR
data for treatment schools; thus, no comparisons were con-
ducted for this measure. Algozzine et al. (2012) examined
the effects of multitiered academic and behavior (i.e.,
SWPBIS) instruction in seven urban elementary schools in
the Southeastern United States. The authors only reported
Gage et al. 3
ODR differences between treatment and control schools at
the beginning of the study and did not report differences at
the end. However, the authors reported on suspensions,
finding no significant difference between the treatment and
control schools. It should be noted that in one additional
SWPBIS RCT, Horner, Sugai, Smolkowski, et al. (2010)
reported ODR data; however, the authors did not compare
ODR rates by treatment conditions.
Three QED studies examined differences in suspensions
and ODR between schools implementing and not imple-
menting SWPBIS. Nelson, Martella, and Marchand-
Martella (2002) evaluated the effects of a comprehensive
school-based program designed to prevent problem behav-
iors and maximize student learning. The program included
five core elements: (a) SWPBIS, (b) one-to-one tutoring in
reading, (c) conflict resolution, (d) a video-based family
management program, and (e) individualized, function-
based behavior intervention plans when necessary. The
study included seven elementary schools in the treatment
group and 28 elementary schools in the comparison group.
All schools were in the same district in the Northeastern
United States. Overall, the authors found significant reduc-
tions in suspensions (d = −0.85) and ODR (d = −0.94).
However, the authors did not report fidelity of implementa-
tion; therefore, it is unclear how well the SWPBIS elements
were implemented. Caldarella, Shatzer, Gray, Young, and
Young (2011) examined the impact of SWPBIS in two mid-
dle schools in the Western United States. Overall, the
authors found no significant differences in the reductions of
ODR between schools at the end of 4 years. Furthermore,
the study compared one school to one school (n = 1 con-
found) and the authors did not report fidelity of implemen-
tation. Flannery et al. (2014) examined the effects of
SWPBIS in 12 high schools in the Northeastern and
Midwestern United States. Overall, the authors found sig-
nificant decreases in ODR across 3 years of implementing
SWPBIS and fidelity of implementation had a significant
effect on ODR reductions.
Overall, results of studies examining the school-level
effects of SWPBIS on suspensions and ODR are mixed.
Furthermore, the small number of schools included in those
studies limits generalization. The largest study only exam-
ined differences in disciplinary exclusions by fidelity levels
and did not include comparisons with business-as-usual
comparison groups (Childs et al., 2016). Taken together, it
is clear that more research is necessary to evaluate the effect
of SWPBIS on disciplinary exclusions.
Purpose
To date, few studies meeting rigorous research design stan-
dards, including RCT and group QED studies with equiva-
lent comparison groups, have been conducted. For example,
What Works Clearinghouse (WWC) sample size criteria for
small or medium and large extent of evidence is n = 350
units of analysis (WWC, 2014). One way to increase the
sample size of efficacy studies is to leverage existing
school-level data made available by state departments of
education. Therefore, to increase the experimental evidence
supporting the implementation of SWPBIS with fidelity on
school-level disciplinary exclusions, we conducted a sec-
ondary analysis of publicly available school-level data in
Georgia and conducted a quasi-experimental analysis using
propensity score matching analysis. Specific research ques-
tions were as follows:
Research Question 1: Is there a statistically significant
difference in the number of disciplinary incidents and in-
school and out-of-school suspensions for schools imple-
menting SWPBIS with fidelity compared with matched
comparison schools?
Research Question 2: Is there a statistically significant
difference in the number of disciplinary incidents and in-
school and out-of-school suspensions for schools imple-
menting SWPBIS at different fidelity levels compared
with matched comparison schools?
Method
Sample
Data from all public schools in the state of Georgia during
the 2015–2016 school year were collected from the Georgia
Department of Education (GaDOE) website. We restricted
the data to only public elementary and intermediate schools,
excluding alternative schools, vocational/technical schools,
middle schools, and high schools because reported training
and fidelity data on SWPBIS included only public elemen-
tary and intermediate schools. The final sample included
1,755 schools. The average school enrollment was 548.5
students (SD = 245.2), and there were equal percentages of
students that were White (40.3%) and Black (39.0%), fol-
lowed by students that were Latino/a (13.8%). On average,
68.5% of the students were considered economically disad-
vantaged and almost 12% of students received special edu-
cation services.
Implementation of SWPBIS in Georgia is supported by
the GaDOE, which facilitates district-level planning and
provides school team training, technical assistance, and
ongoing coaching to SWPBIS district coordinators to build
capacity and support the SWPBIS process. SWPBIS imple-
mentation occurs in districts where active support through a
district leadership team, a district action plan, and district
coordinator has been established. The GaDOE website
reports the names of schools receiving SWPBIS training and
support, and three levels of fidelity of implementation:
Installing, defined primarily as fidelity below 70% on the
Benchmarks of Quality (BoQ); Emerging, defined as fidelity
4 Journal of Positive Behavior Interventions 00(0)

Table 1.  Descriptive Statistics for Full Sample and Propensity Score–Matched Samples.

Treatment
Full sample PSM comparison schools
(n = 1,418) schools (n = 119) (n = 119) Equivalence

School characteristics M SD M SD M SD Full PSM

Language 36.2 17.9 36.6 15.9 35.9 13.6 −0.02 −0.05
Math 37.2 19.0 40.1 17.1 39.2 14.8 0.11 −0.06
Science 34.1 18.0 37.3 16.6 36.6 14.3 0.14 −0.05
Social studies 31.4 18.0 32.7 16.9 32.3 13.7 0.05 −0.03
% Black 38.3 30.7 29.4 27.9 32.7 27.1 −0.18 0.12
% White 13.5 15.5 47.3 29.4 44.1 29.6 1.80 −0.11
% Latino/a 41.3 29.2 17.3 22.3 16.2 15.8 −0.88 −0.06
% Economically disadvantaged 67.3 28.6 69.0 27.6 68.9 23.6 0.06 0.00
% Students with disabilities 11.8 4.9 11.5 3.5 11.2 3.1 −0.13 −0.09
% Students limited English proficient 8.4 13.1 12.0 19.1 12.2 15.3 0.29 0.01
Enrollment 553.9 254.4 528.2 253.6 528.9 170.7 −0.10 0.00

Note. PSM = propensity score–matched. Equivalence is the standardized mean difference between the full sample and treatment (Full) and the PSM
comparison schools and the treatment schools (PSM).

between 70% and 85% on the BoQ; and Operational, defined
as BoQ above 85%. In this study, we focused exclusively on
schools implementing SWPBIS with fidelity (Emerging and
Operational) and excluded schools (n =218) that were
Installing as they were not implementing Tier 1 prevention
efforts with fidelity but had received training. A total 119
schools implemented SWPBIS with fidelity during the
2015–2016 school year. The average enrollment for schools
implementing SWPBIS with fidelity (i.e., treatment schools)
was 528.9 students (SD = 170.7) and the majority of students
were White (44.1%), followed by students that were Black
(32.6%) and students that were Latino/a (16.2%). On aver-
age, 68.9% of the students were economically disadvantaged
and 11.2% of students received special education services.
Demographic characteristics for all schools by fidelity
groups are presented in Table 1.
Measures
BoQ.  The BoQ (Childs, Kincaid, & George, 2011; Cohen,
Kincaid, & Childs, 2007; Kincaid, Childs, & George, 2005,
2010) is a self-report measure used to assess the implemen-
tation fidelity of SWPBIS at the Tier 1/universal level. The
BoQ consists of 53 items rated on a 3-point Likert-type
scale (i.e., In Place, Needs Improvement, and Not in Place).
Prior psychometric evidence suggests that the BoQ demon-
strates strong internal consistency (overall α = .96), inter-
rater reliability (r = .87), and test–retest reliability (r = .94).
The 53 items are organized under 10 subscales reflecting
the essential components of Tier 1 implementation (e.g.,
faculty commitment, expectations and rule developed,
classroom systems). Scores for the BoQ are scaled as the
percentage of points earned out of the total possible points
(107), with scores 70% or above considered implementing
with fidelity. The BoQ scores are collected each year in
April or May for all schools implementing SWPBIS in
Georgia.
School-level covariates
Student demographics.  We included seven student demo-
graphic covariates in the final dataset. First, we captured the
total student enrollment for each school, and the percentage
of students in each school that were categorized as White,
Black, or Latino/a. Next, we captured the percentage of stu-
dents in each school that was categorized as economically
disadvantaged, defined as the number of students eligible
to receive free- or reduced-price meals (as reported to the
GaDOE in the October Nutrition Count) divided by the total
school enrollment (as reported by the October Full-Time
Equivalency[FTE] count). Last, we collected the percent-
age of students with disabilities, defined as any student
receiving special education services, and the percentage of
students categorized as limited English proficient (LEP) by
their school.
Georgia Milestones Assessment System. The Georgia
Milestones Assessment System measures how well students
have learned the knowledge and skills outlined in the state-
adopted content standards in English Language Arts, Math-
ematics, Science, and Social Studies. Students in Grades 3
through 8 take an end-of-grade assessment in English Lan-
guage Arts and Mathematics while students in Grades 5 and
8 are also assessed in Science and Social Studies. Students
receive a scale score and an Achievement Level designation
based on their total test performance in each content area.
Achievement levels are as follows: Beginning Learner,
Gage et al. 5
Developing Learner, Proficient Learner, and Distinguished
Learner. We captured the percentage of students at the Pro-
ficient Learner and above levels for each school across the
four subject tests because this level indicates grade-level
performance.
Outcome variables
Behavioral incidents.  The GaDOE collects the frequency
of 36 different behavioral incidents that result in an office
disciplinary referral for all schools in Georgia. The incidents
are operationally defined by GaDOE Pursuant to O.C.G.A.
§ 20-2-740 in a discipline matrix designed to provide guid-
ance for schools and districts (Georgia Department of Edu-
cation, 2017). During the 2015–2016 school year, 33% of
all reported incidents were for incivility, defined as insub-
ordination or disrespect to staff members or other students,
which includes but is not limited to refusal to follow school
staff member instructions, use of vulgar or inappropri-
ate language, and misrepresentation of the truth. Another
21% of incidents were for disorderly conduct, defined as
any act that substantially disrupts the orderly conduct of a
school function, substantially disrupts the orderly learning
environment, or poses a threat to the health, safety, and/
or welfare of students, staff, or others (includes disruptive
behaviors on school buses). We summed all 36 behavioral
incidents together because (a) occurrences were infrequent
across many of the incidents (e.g., gang-related incident)
and (b) we were interested in the impact of SWPBIS on
disciplinary incidents broadly as a proxy for ODR.
School suspensions.  The GaDOE also collects frequency
(i.e., number, not length) data on disciplinary outcomes,
including in- and out-of-school suspension. In-school sus-
pensions are defined by GaDOE as the temporary removal
of a student from his or her regular classroom(s) for at least
half a school day. The student remains under the direct
supervision of school personnel (direct supervision means
school personnel are physically in the same location as stu-
dents under their supervision; Georgia Department of Edu-
cation, 2017). Out-of-school suspension is defined simply
as the removal of a student from the school for at least one
school day.
Data Analysis
To address the primary research questions in this study, we
conducted a QED comparing schools implementing SWPBIS
with fidelity to propensity score–matched comparison
schools not implementing SWPBIS.
Propensity score matching.  Propensity score matching (PSM)
methods are designed to reduce bias in treatment effect esti-
mates in experimental design studies that do not have random
assignment of participants to conditions (i.e., Leite, 2017). A
propensity score is defined as the conditional probability of
treatment assignment based on all available covariates
(Rosenbaum & Rubin, 1983) and can be used for one-to-one
matching treatment to comparison schools. The value of
PSM is that a covariate equivalent comparison group can be
matched to a treatment group, meeting established standards
for high-quality QED research proposed by the WWC (2014)
evidence standards, and that treatment estimates have been
found to be as accurate as those from RCT studies (Fortson,
Verbitsky-Savitz, Kopa, & Gleason, 2012).
Following procedures outlined by Leite (2017), we esti-
mated propensity scores using logistic regression and all
available school-level covariates (see Table 1). Specifically,
we created a dichotomous variable for all schools. Schools
that implemented SWPBIS with fidelity were coded as 1,
while all other schools were coded as a 0. Next, we esti-
mated the predicted probability (p), or propensity score,
that a school was in the treatment or control group based on
the included covariates, log(p / [1 – p]). Then we used the
estimated propensity scores to match schools using the one-
to-one optimal matching method (Rosenbaum, 1989),
which minimizes global propensity score distance between
treatment and comparison schools. Essentially, the one-to-
one matching procedure identifies a perfect match school
for each treatment school so that the treatment and compari-
son schools were equivalent, or the same, on all of the
included covariates. In an RCT, any difference between
treatment and comparison schools is assumed to be random
error, while in PSM, differences between the two groups on
school characteristics are equivalent, ensuring that any
treatment results are based on the intervention, not any
other potential confound. The one-to-one optimal matching
algorithm was conducted using the matchit (Ho, Imai, King,
Stuart, & Whitworth, 2017) and optmatch (Hansen,
Fredrickson, Fredrickson, Rcpp, & Rcpp, 2016) packages
in R (R Core Team, 2013). To confirm covariate equiva-
lence, we calculated standardized mean difference effect
sizes (d), where equivalence is defined as d < .25 standard
deviations (WWC, 2014).
Estimation of treatment effects.  The three primary outcome
variables (i.e., behavioral incidents, in-school suspensions,
and out-of-school suspensions) were all scaled as counts;
therefore, modeling of treatment effects relied upon Poisson
regression to accurately estimate treatment effects given
their distributional characteristics. Although the PSM
model, or design part of the analysis (Rubin, 2007), con-
trolled for all potential available confounds on the treatment
effect (Leite, 2017), we controlled for covariates with stan-
dardized mean differences greater than .05 (WWC, 2014).
As noted, we estimated six Poisson regression models:
three treatment-on-the-treated (ToT) models and three mod-
els examining differences by fidelity levels using the fol-
lowing formulas:
6 Journal of Positive Behavior Interventions 00(0)
log ( λ ) = β0 + β1Treatment + β2 % White + β3 %Black
D + β6 Language
+ β4 %Hispanic + β5 %SPED
+ β7 Math + β8Science.
log ( λ ) = β0 + β1Fidelity Levels + β2 % White + β3 %Black
+ β4 %Hispanic + β5 %SPED + β6 Language
+ β7 Math + β8Science.
Effect sizes.  To increase interpretation of the treatment effects,
we calculated standardized mean difference (g) effect sizes
controlling for covariates with equivalence values greater the
.05 standard deviation units following WWC (2014) stan-
dards. Our interpretation of the effect sizes is based on
Cohen’s “rule of thumb,” where g = .20 is small, g = .50 is
medium, and g = .80 is large.
Results
Establishing Equivalence
We estimated a PSM model using available school-level
covariates for all general education elementary and inter-
mediate public schools in Georgia to identify an equivalent
comparison group. Table 1 provides the average value for
all available covariates for schools implementing SWPBIS
with fidelity, all possible comparison schools, and the PSM
comparison schools. We examined equivalence between the
samples by calculating the standardized mean difference (g)
across the school-level covariates and defined equivalence
as differences less than .25 standard deviation units (WWC,
2014). For the comparison between the full sample and the
treatment schools, three covariates were not equivalent (i.e.,
% of White students, % of Latino/a students, % of students
LEP) and six covariates were at or above .05 standard devi-
ation units. For the PSM samples, all covariates were less
than .25 standard deviation units, seven covariates were
greater than or equal to .05 standard deviation units, and
four covariates were less than .05 standard deviation units.
Overall, equivalence was established for all covariates for
the PSM comparison and treatment schools.
Treatment Effects
We estimated six Poisson regression models to determine
whether or not there was a statistically significant differ-
ence in the frequency of in- and out-of-school suspensions
and disciplinary incidents for schools implementing
SWPBIS and matched comparison schools. Unlike ordinary
least squares (OLS) regression, the coefficients are not the
unit change in the dependent variable for each unit change
in the treatment variable, but instead, the Poisson regression
coefficient is interpreted as follows: For 1 unit change in the
predictor variable, the difference in the logs of expected
counts is expected to change by the respective coefficient.
Therefore, to determine the strength of the relationship rela-
tive to the scaling, the coefficient (β) becomes the exponent
of e, and is subtracted from 1 and multiplied by 100 to iden-
tify the relative percentage change in dependent variable
counts for the treatment group. This calculation was con-
ducted in Microsoft Excel for each treatment coefficient
using the following function:
1 − (exp(β)) .
The first series of models, presented in Table 2, exam-
ined the treatment effects for the 119 schools implementing
SWPBIS with fidelity (i.e., BoQ ≥ 70%). There were statis-
tically significantly fewer in-school and out-of-school sus-
pensions, and disciplinary exclusions in schools
implementing SWPBIS with fidelity after controlling for all
covariates with equivalence values greater than .05 standard
deviation units. Schools implementing SWPBIS with fidel-
ity had 55.1% fewer out-of-school suspensions, 79.8%
fewer in-school suspensions, and 58.9% fewer disciplinary
incidents than the PSM comparison schools.
The second series of models, presented in Table 3,
focused on differences by fidelity. Specifically, the models
examined the differences between schools implementing
SWPBIS with BoQ scores between 70% and 84%
(Emerging) and schools with BoQ scores equal to or greater
than 85% (Operational). Little difference was found
between fidelity levels for in-school suspensions. Schools
implementing with higher levels of fidelity had fewer out-
of-school suspensions and disciplinary incidents than
schools implementing below 85% on the BoQ. However,
the differences between both fidelity groups were signifi-
cantly less than PSM comparison schools.
Last, we calculated standardized mean difference effect
sizes (g) to estimate the magnitude of effect SWPBIS has
on the three outcomes, out-of-school suspensions, in-school
suspensions, and disciplinary incidents. Following WWC
requirements, we adjusted the means for the six covariates
with equivalence values > .05 standard deviation units. The
covariate adjusted means, standard deviations, and effect
sizes are presented in Table 4. Across the three outcomes,
all effect sizes were medium and the largest effect size was
found for in-school suspensions.
Discussion
SWPBIS has been recommended as an evidence-based
approach to reduce the need for and use of school suspen-
sion (American Academy of Pediatrics Council on School
Health, 2013), yet large-scale experimental evidence has
been limited. To address this gap, we conducted a state-
level analysis on the treatment effects of SWPBIS on dis-
ciplinary incidents and school suspensions. Overall, we
Gage et al. 7

Table 2.  Treatment Effect Poisson Regression Models.

Out-of-school suspension In-school suspension Disciplinary incidents

Parameter β SE β SE β SE
Intercept 3.79*** 0.259 2.86*** 0.221 3.74*** 0.149
SWPBIS −0.80*** 0.020 −1.60*** 0.021 –0.89*** 0.011
% White 0.00 0.002 0.01*** 0.002 0.01*** 0.001
% Black 0.01*** 0.003 0.01*** 0.002 0.02*** 0.001
% Latino/a 0.00 0.003 0.01* 0.002 0.01*** 0.002
% SPED 0.05*** 0.003 0.09*** 0.002 0.05*** 0.002
% Proficient: Math −0.05*** 0.002 −0.12*** 0.002 –0.07*** 0.001
% Proficient: Language Arts 0.03*** 0.002 0.08*** 0.002 0.04*** 0.001
% Proficient: Science 0.01*** 0.002 0.04*** 0.001 0.03*** 0.000

Note. SWPBIS = a dichotomous indicator for school implementing school-wide positive behavior intervention and supports; SPED = special education.
*p < .05. ***p < .001

Table 3.  Treatment Effect Poisson Regression Models by Fidelity Levels.

Out-of-school
suspension In-school suspension Disciplinary incidents

Parameter β SE β SE β SE
Intercept 3.51*** 0.263 2.87*** 0.222 3.53*** 0.150
SWPBIS: Emerging −0.63*** 0.028 −1.62*** 0.033 −0.75*** 0.016
SWPBIS: Operational −0.92*** 0.025 −1.59*** 0.024 −0.98*** 0.014
% White 0.00 0.002 0.01*** 0.002 0.01*** 0.001
% Black 0.02*** 0.003 0.01*** 0.002 0.02*** 0.001
% Latino/a 0.00 0.003 0.01* 0.002 0.01*** 0.002
% SPED 0.05*** 0.003 0.09*** 0.002 0.05*** 0.002
% Proficient: Math −0.05*** 0.002 −0.12*** 0.002 −0.07*** 0.001
% Proficient: Language Arts 0.03*** 0.002 0.08*** 0.002 0.04*** 0.001
% Proficient: Science 0.01*** 0.002 0.04*** 0.001 0.03*** 0.001

Note. SWPBIS = School-Wide Positive Behavior Interventions and Supports and fidelity levels was factored so that control schools are the reference
group for the emerging and operational groups; SPED = special education.
*p < .05. ***p < .001

Table 4.  Standardized Mean Difference Effect Sizes for fidelity. The effect sizes suggest that differences were, on
Treatment and PSM Comparison Schools. average, almost 0.50 standard deviation units.
These findings extend prior research by evaluating treat-
PSM
SWPBIS comparison ment effects on a much larger sample of schools and utilizes
schools schools a QED that could meet the WWC evidence standards with
reservations. Specifically, we included a total of 238 schools,
Outcome M SD M SD g a sample almost 3 times that of all prior experimental studies
OSS 29.9 34.2 69.2 95.8 −0.54 on suspensions (Gage et al., in press), and established base-
ISS 24.6 55.7 123.1 188.2 −0.71 line equivalence on relevant covariates. Perhaps the most
Disciplinary incident 92.1 97.5 232.6 291.2 −0.64 directly comparable study to ours is the Bradshaw et al.
(2010) RCT. They randomly assigned 37 elementary schools
Note. PSM = propensity score–matched; SWPBIS = a dichotomous to treatment (n = 21) and comparison (n = 16) groups and
indicator for school implementing school-wide positive behavior
intervention and supports; OSS = out-of-school suspension; ISS = in- trained schools in the treatment group to implement univer-
school suspension. sal SWPBIS with fidelity. However, after 4 years of imple-
mentation, they found no significant difference in school
found statistically significantly fewer behavioral incidents suspensions between treatment and comparison groups. Two
and suspensions in schools implementing SWPBIS with possible reasons for the difference in findings were (a) that
8 Journal of Positive Behavior Interventions 00(0)
schools in the Bradshaw et al. study had very few suspen-
sions and (b) the school-level sample size was not large
enough to estimate a meaningful treatment effect. On aver-
age, the treatment schools suspended ~30 students, while the
comparison schools suspended an average of ~17 students
across the 4 years of implementation. The authors reported
suspensions as the percentage of students and analyzed the
differences using a nonparametric test, presumably due to
the scaling of their outcome (i.e., percentage of students) and
the small sample size (n = 37). In our study, we were able to
(a) collect the exact number of suspensions from each school
and (b) estimate an effect with a much larger sample size,
resulting in a significant treatment effect.
With regard to ODRs, the most comparable study used a
QED and was conducted by Nelson et al. (2002). Nelson
and colleagues trained seven elementary schools to imple-
ment SWPBIS across all three tiers and found significant
and large differences in the number of ODR between the
treatment and comparison students, similar to the results of
this study. However, Nelson et al. trained schools to imple-
ment all three tiers of prevention and intervention, while the
schools in this study were only evaluated on their universal
implementation as measured by the BoQ. Some of the
schools may have been implementing additional tiers, but
no addition information beyond universal implementation
was available. The difference in effect sizes in our study
may be the additional impact adding Tiers 2 and 3 can have
on ODR. Unfortunately, Nelson and colleagues did not
report fidelity of implementation; therefore, future research
is needed to evaluate any additive effect on ODR for imple-
menting Tiers 2 and 3 with fidelity.
In addition to overall treatment effects, we examined dif-
ferences by fidelity levels as defined by the GaDOE.
Specifically, we examined whether higher levels of imple-
mentation fidelity, defined as 85% or greater on the BoQ,
resulted in fewer behavioral incidents and suspensions.
Overall, both groups reported statistically significant fewer
incidents and suspensions than the PSM comparison schools.
However, for out-of-school suspensions and behavioral inci-
dents, schools implementing with higher levels of fidelity
had larger treatment effects, suggesting that fidelity has a
direct relationship with incidents and out-of-school suspen-
sions. This finding is encouraging as it suggests that the
more components of Tier 1 a school implements, the greater
the impact on behavioral incidents and the lower likelihood
a student receives an out-of-school suspension. This finding
corroborates the results of prior longitudinal evaluations of
SWPBIS. Childs et al. (2016) found that as fidelity of imple-
mentation, as measured by the BoQ, increased in Florida
schools, both suspensions and ODR decreased. Similar
results were found by Simonsen et al. (2012) for suspen-
sions. Therefore, there appears to be a relation between
fidelity of implementation and ODR and suspensions, with
greater fidelity resulting in fewer exclusions.
Limitations
Although all efforts were made to ensure the highest quality
QED study, a number of limitations necessitate mention.
First, this study relied solely on administrative data, includ-
ing reporting of fidelity of implementation. There is no way
to independently confirm the reliability of the BoQ scores
and the fidelity levels reported on the GaDOE website.
Relatedly, there is no way to confirm the reliability of the
outcome measures (i.e., discipline incidents and suspen-
sions). Second, although PSM created an equivalent com-
parison group, there are other potential confounds we were
unable to match schools on. The goal of PSM is to include
all possible school covariates, but not all potential con-
founders were available. For example, some schools may
have been under state investigation for disproportionate
suspension rates or have superintendents or principals with
new goals for decreasing suspensions. Future research may
consider additional covariates related to school policies cor-
related with the likelihood of school suspensions. Third, the
suspension outcome did not disaggregate suspensions by
the number of days students received in- and out-of-school
suspensions. Therefore, we cannot confirm whether or not
there was a differential effect on suspension length. Fourth,
fidelity of implementation was based on the BoQ, which is
typically completed by the school and their SWPBIS coach.
The addition of an external measure, such as the SET, would
increase the reliability of the fidelity score. Last, this study
is a QED relying on administrative data. There is no way to
confirm whether or not data entry errors could have been
made. Future studies should consider utilizing RCT designs
in which they assign schools at random to SWPBIS and
control conditions to increase the validity of the research
findings.
Implications
In this study, we evaluated the efficacy of SWPBIS imple-
mented with fidelity on behavioral incidents and school
suspensions and found significant reductions for both out-
comes. Our results have important implications for both
researchers and practitioners. First, the significant and mod-
erate treatment effect needs to be replicated in other states
and settings, controlling for additional confounds and
including additional outcomes and populations (e.g., stu-
dents receiving special education services). We believe that
this study provides a template for future state-level analyses
that can be replicated broadly. Second, the results suggest
that state-level initiatives, like Georgia’s, can (a) effectively
train districts and schools to implement SWPBIS with fidel-
ity in large numbers of schools and (b) that the training and
subsequent implementation can have a meaningful impact
on disciplinary incidents and suspensions. Finally, we
believe the study results provide evidence that fidelity of
Gage et al. 9
implementation of SWPBIS is a critical component of
effectiveness and that schools should strive to implement
SWPBIS universal practices with fidelity to significantly
decrease behavior incidents and suspensions.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) received no financial support for the research,
authorship, and/or publication of this article.
References
Algozzine, B., Wang, C., White, R., Cooke, N., Marr, M. B.,
Algozzine, K., & Zamora Duran, G. (2012). Effects of
multi-tier academic and behavior instruction on diffi-
cult-to-teach students. Exceptional Children, 79, 45–64.
doi:10.1177/001440291207900103
American Academy of Pediatrics Council on School Health.
(2013). Out-of-school suspension and expulsion. Pediatrics,
131, e1000–e1007. doi:10.1542/peds.2012-3932
Arcia, E. (2006). Achievement and enrollment status of sus-
pended students: Outcomes in a large, multicultural school
district. Education and Urban Society, 38, 359–369.
doi:10.1177/0013124506286947
Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J.
(2009). Altering school climate through school-wide posi-
tive behavioral interventions and supports: Findings from a
group-randomized effectiveness trial. Prevention Science, 10,
100–115.
Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining
the effects of schoolwide positive behavioral interventions
and supports on student outcomes: Results from a random-
ized controlled effectiveness trial in elementary schools.
Journal of Positive Behavior Interventions, 12, 133–148.
doi:10.1177/1098300709334798
Caldarella, P., Shatzer, R. H., Gray, K. M., Young, K. R., & Young, E. L.
(2011). The effects of school-wide positive behavior supports on
middle school climate and student outcomes. Research in Middle
Level Education, 35(4), 1–14. Retrieved from http://www.amle.
org/ServicesEvents/ResearchinMiddleLevelEducationOnline/
tabid/173/Default.aspx
Childs, K. E., Kincaid, D., & George, H. P. (2011). The revised
school-wide PBS Benchmarks of Quality (BoQ). OSEP
Technical Assistance Center on Positive Behavioral
Interventions and Supports. Retrieved from http://www.pbis.
org/evaluation/evaluation_briefs/mar_11_(1).aspx
Childs, K. E., Kincaid, D., George, H. P., & Gage, N. A. (2016).
The relationship between school-wide implementation of
positive behavior intervention and supports and student dis-
cipline outcomes. Journal of Positive Behavior Interventions,
18, 89–99.
Cohen, R., Kincaid, D., & Childs, K. (2007). Measuring school-
wide positive behavior support implementation: Development
and validation of the Benchmarks of Quality (BoQ). Journal
of Positive Behavior Interventions, 9, 203–213. doi:10.1177/1
0983007070090040301
Crone, D. A., Hawken, L. S., & Horner, R. H. (2010). Responding
to problem behavior in schools: The behavior education pro-
gram. New York, NY: Guilford Press.
Flannery, K. B., Fenning, P., Kato, M. M., & McIntosh, K. (2014).
Effects of school-wide positive behavioral interventions and
supports and fidelity of implementation on problem behavior
in high schools. School Psychology Quarterly, 29, 111–124.
doi:10.1037/spq0000039
Fortson, K., Verbitsky-Savitz, N., Kopa, E., & Gleason, P. (2012).
Using an experimental evaluation of charter schools to test
whether nonexperimental comparison group methods can
replicate experimental impact estimates (NCEE Technical
Methods Report 2012-4019). Washington, DC: National Center
for Education Evaluation and Regional Assistance, Institute of
Education Sciences, U.S. Department of Education, 2012.
Freeman, J., Simonsen, B., McCoach, D. B., Sugai, G., Lombardi,
A., & Horner, R. (2016). Relationship between school-
wide positive behavior interventions and supports and aca-
demic, attendance, and behavior outcomes in high schools.
Journal of Positive Behavior Interventions, 18, 41–51.
doi:10.1177/1098300715580992
Gage, N. A., Leite, W., Childs, K., & Kincaid, D. (2017). Average
treatment effect of school-wide positive behavioral interven-
tions and supports on school-level academic achievement in
Florida. Journal of Positive Behavior Interventions, 19, 158–
167. doi:10.1177/1098300717693556
Gage, N. A., Whitford, D. K., & Katsiyannis, A. (in press). A review
of schoolwide positive behavior interventions and supports as
a framework for reducing disciplinary exclusions. Journal of
Special Education. doi: 10.1177/0022466918767847
Georgia Department of Education. (2017). Discipline matrix
rationale. Retrieved from www.gadoe.org/Curriculum-
Instruction-and-Assessment/Curriculum-and-Instruction/
Documents/DisciplineMatrixRationale.pdf
Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood,
C., & Innocenti, M. S. (2005). Quality indicators for group
experimental and quasi-experimental research in special edu-
cation. Exceptional Children, 71, 159–164.
Hansen, B. B., Fredrickson, M., Fredrickson, M. M. M., Rcpp,
L., & Rcpp, I. (2016). optmatch (R package version 0.9-6).
Retrieved from https://cran.r-project.org/web/packages/opt-
match/
Ho, D., Imai, K., King, G., Stuart, E., & Whitworth, A. (2017).
MatchIt: Nonparametric preprocessing for parametric causal
inference. Journal of Statistical Software, 42(8), 1–28.
Retrieved from http://www.jstatsoft.org/v42/i08/
Horner, R. H., Sugai, G., & Anderson, C. M. (2010). Examining
the evidence base for school-wide positive behavior support.
Focus on Exceptionality, 42, 1–14.
Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato,
J., Todd, A. W., . . . Esperanza, J. (2010). A randomized,
wait-list controlled effectiveness trial assessing school-
wide positive behavior support in elementary schools.
Journal of Positive Behavior Interventions, 11, 133–144.
doi:10.1177/1098300709332067
Kincaid, D., Childs, K., & George, H. P. (2005). School-wide
Benchmarks of Quality (BoQ). Unpublished instrument,
10 Journal of Positive Behavior Interventions 00(0)
Department of Child & Family Studies, University of South
Florida, Tampa. Retrieved from http://flpbs.fmhi.usf.edu/
ProceduresTools.cfm
Kincaid, D., Childs, K., & George, H. P. (2010). School-wide
Benchmarks of Quality (BoQ). Unpublished instrument,
Department of Child & Family Studies, University of South
Florida, Tampa. Retrieved from http://flpbs.fmhi.usf.edu/
ProceduresTools.cfm
Kupchik, A., & Catlaw, T. J. (2015). Discipline and participation:
The long-term effects of suspension and school security on
the political and civic engagement of youth. Youth & Society,
47, 95–124. doi:10.1177/0044118X14544675
Leite, W. (2017). Propensity score methods using R. Los Angeles,
CA: SAGE.
Losen, D., Hodson, C., Keith, M. A., Morrison, K., & Belway,
S. (2015). Are we closing the school discipline gap?
UCLA Center for Civil Rights Remedies. Retrieved from
https://www.civilrightsproject.ucla.edu/resources/projects/
center-for-civil-rights-remedies/school-to-prison-folder/
federal-reports/are-we-closing-the-school-discipline-gap/
AreWeClosingTheSchoolDisciplineGap_FINAL221.pdf
Mitchell, B. S., Stormont, M., & Gage, N. A. (2011). Tier two
interventions implemented within the context of a tiered pre-
vention framework. Behavioral Disorders, 36, 241–261.
Mowen, T., & Brent, J. (2016). School discipline as a turn-
ing point: The cumulative effect of suspension on arrest.
Journal of Research in Crime & Delinquency, 53, 628–653.
doi:10.1177/0022427816643135
Mowen, T. J., & Manierre, M. J. (2015). School security measures
and extracurricular participation: An exploratory multi-level
analysis. British Journal of Sociology of Education, 38, 344–
363. doi:10.1080/01425692.2015.1081091
Nelson, J. R., Martella, R. M., & Marchand-Martella, N. (2002).
Maximizing student learning: The Effects of a comprehen-
sive school-based program for preventing problem behaviors.
Journal of Emotional and Behavioral Disorders, 10, 136–
148. doi:10.1177/10634266020100030201
Noltemeyer, A. L., Ward, R. M., & McLoughlin, C. (2015).
Relationship between school suspension and student out-
comes: A meta-analysis. School Psychology Review, 44, 224–
240. doi:10.17105/spr-14-0008.1
R Core Team. (2013). R: A language and environment for statisti-
cal computing. Vienna, Austria: R Foundation for Statistical
Computing. Available from http://www.R-project.org/
Rosenbaum, P. R. (1989). Optimal matching for observational
studies. Journal of the American Statistical Association, 84,
1024–1032.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the
propensity score in observational studies for causal effects.
Biometrika, 70, 41–55.
Rubin, D. B. (2007). The design versus the analysis of observa-
tional studies for causal effects: Parallels with the design of
randomized trials. Statistics in Medicine, 26, 20–36.
Sadler, C., & Sugai, G. (2009). Effective behavior and instruc-
tional support: A district model for early identification and
prevention of reading and behavior problems. Journal of
Positive Behavior Interventions, 11, 35–46.
Simonsen, B., Eber, L., Black, A. C., Sugai, G., Lewandowski,
H., Sims, B., & Meyers, D. (2012). Illinois statewide posi-
tive behavioral interventions and supports: Evolution
and impact on student outcomes across years. Journal of
Positive Behavior Interventions, 14, 5–16. doi:10.1177/1098
300711412601
Sugai, G., & Horner, R. H. (2009). Defining and describing school-
wide positive behavior support. In W. Sailor, G. Dunlap, G.
Sugai, & R. Horner (Eds.), Handbook of positive behavior
support (pp. 307–326). New York, NY: Springer.
Sugai, G., Simonsen, B., Bradshaw, C., Horner, R., & Lewis, T. J.
(2014). Delivering high quality school wide positive behavior
support in inclusive schools. In J. McLeskey, N. L. Waldron,
F. Spooner, & B. Algozzine (Eds.), Handbook of effective
inclusive schools: Research and practice (pp. 306–321). New
York, NY: Routledge.
U.S. Department of Education. (2016). 2013-2014 civil rights data
collection: A first look. Washington, DC: U.S. Department of
Education Office for Civil Rights. Retrieved from https://
www2.ed.gov/about/offices/list/ocr/docs/2013-14-first-look.
pdf
Waasdorp, T. E., Bradshaw, C. P., & Leaf, P. J. (2012). The
impact of schoolwide positive behavioral interventions and
supports on bullying and peer rejection: A randomized con-
trolled effectiveness trial. Archives of Pediatrics & Adolescent
Medicine, 166, 149–156.
What Works Clearinghouse. (2014). What Works Clearinghouse
procedures and standards handbook (Version 3.0). Author:
Washington, DC. Retrieved from https://ies.ed.gov/ncee/
wwc/Docs/referenceresources/wwc_procedures_v3_0_stan-
dards_handbook.pdf