Education Research Journal Vol. 1(3), pp.

37-42, August 2011

Available online at http://www.resjournals.com/ARJ
ISSN: 2026 6332 2011 International Research Journals

Full l Length Research paper

Assessing Power of Structural Equation Modeling

Studies: A Meta-Analysis
Fatma KAYAN FADLELMULA
Middle East Technical University, Faculty of Education Department of Elementary Education
06531, Ankara, TURKEY
Emil:kayan@metu.edu.tr
Abstract

A meta-analysis was performed to determine the statistical power of research studies utilizing
Structural Equation Modeling as the primary analysis technique. An extensive literature review was
conducted in the Turkish Social Sciences Database, for journals published in the field of education. The
database covered 34 journals in this field. After a full text review, 12 studies, published from 2007 to
2010, were found suitable for analysis. Power was calculated using SAS; consideration was given to,
indicated degrees of freedom, significance level, sample size, and root-mean-square error of
approximation. Results revealed that one fourth of the studies achieved power less than 0.50.
Considering that rejecting a false null hypothesis with a power of 0.50 is the same as guessing the
outcome by chance, the reported findings of these studies would possess relatively little practical
significance. Recommendations are made for future research and practice.
Keywords: Structural equation modeling, power, meta-analysis
the structural model (Kelloway, 1998).
Nowadays, SEM has become increasingly popular
INTRODUCTION
among researchers from many different disciplines.
Interest in SEM is evident by the growing number of
Structural equation modeling (SEM) is a statistical
software programs developed to apply SEM (such as
technique that takes a hypothesis testing approach to
AMOS, EQS, LISREL, and Mplus), numerous graduate
the analysis of a structural theory (Raykov and
level courses and continuing education workshops
Marcoulides, 2006). It is used for both construct
prepared for explaining SEM, and many published
validation and theory development (Pedhazur and
Pedhazur, 1991). Procedurally, structural equation
modeling works with a correlation or a covariance data
matrix, derived from a set of observed or latent
variables (Kunnan, 1998), and it attempts to explain the
patterns of covariance among the variables included in

empirical works where the researchers describe their

SEM results (Kline, 2011). This is mostly due to the fact
that most of the data analytical techniques, such as
exploratory factor analysis, confirmatory factor analysis,
multiple regression, path analysis, and several analysis
of variance as well as multivariate analysis of variance,
are special cases of structural equation modeling
(Sapp, 2006). Basically, SEM combines those statistical

techniques into one comprehensive analysis (Kunnan,

1998). However, it should be noted that although SEM
is a powerful multivariate technique, the conclusions
reached from SEM may not be as strong as those
obtained from true experimental research designs,
specially regarding the threats to internal validity (Sapp,
2006, p.47).
Most applications of structural equation modeling
include five stages. These are model specification,
model identification, model estimation, model testing,
and model

38

Educ. Res. J.

Table 1.Model Fit Indices and Their Criterion

Fit Indices

Criterion
Non-significant

Chi-Squared ( )

GFI>0.90

Goodness of Fit Index (GFI)

AGFI>0.90

Adjusted Goodness of Fit Index (AGFI)

Root Mean Square Error of Approximation (RMSEA)

0.05<RMSEA<0.10 (moderate
fit)
RMSEA<0.05
(a very good fit)

Root Mean Square Residual (RMR)

RMR<0.05

Standardized Root Mean Square Residual

S-RMR<0.05

(S-RMR)

NFI>0.90

Normed Fit Index (NFI)

NNFI>0.90

Non-Normed Fit Index (NNFI)

Comparative Fit Index (CFI)

CFI>0.90

Incremental Fit Index (IFI)

IFI>0.90

Relative Fit Index (RFI)

RFI>0.90

modification, or pruning (Bollen and Long,

1993). Among these stages, model testing
refers to the evaluation of whether the
theoretical model is supported by the sample
data and is crucial in the analysis. In the
literature, there are a number of fit indices that
provide a statistical assessment of what it
means to say the model fits the data (Jreskog
and Srbom, 1993). The most commonly used
2

ones are the chi square ( ), the goodness of

fit index (GFI), the adjusted goodness of fit
index (AGFI), and the root mean squared error
of approximation (RMSEA) (Schumacker and
Lomax, 2004).
Table 1 summarizes a number of model fit
indices and their criterion used in SEM
2

analyses. In particular, chi square ( ) is a

measure for overall fit of the model to the data

(Jreskog and Srbom, 1993). A significant

value indicates that the observed and
estimated variance-covariance matrices differ,
2

whereas a nonsignificant value indicates

that there is no significant difference. Hence, a
nonsignificant

value

with

associated

degrees of freedom implies that the model fits

the data well (Kelloway, 1998). GFI measures
how much better the model fits as compared to
no model at all (Jreskog and Srbom, 1993).
The measure of GFI ranges from 0 (no fit) to 1
(perfect fit), with values exceeding 0.90,
indicating a good fit to the data (Kelloway,
1998). In a similar vein, AGFI adjusts GFI for
degrees of freedom; values exceeding 0.90
indicate a good fit to data (Schumacker and
Lomax, 2004).
In addition, RMR is the square root of the
mean of the squared discrepancies between
the implied and observed covariance matrices
(Schumacker and Lomax, 2004). It is used to
compare the fit of two different models with the

same data. The lowest bound of RMR is 0 and

low values are taken to indicate good fit
(Kelloway, 1998). However, this index is
sensitive to the scale of measurement of the
model variables. Therefore, it is difficult to
determine what a low value actually is. So,
standardized root mean square residual (SRMR) is used as an alternative index, which is
a standardized summary of the average
covariance residuals. Covariance residuals are
the differences between the observed and
estimated covariance. Values of the S-RMR
range from 0 (perfect fit) to 1 (poor fit). As the
difference between the observed and predicted
covariances increases, the value of the S-RMR
increases as well. The values of S-RMR less
than 0.05 indicate a good fit to the data;
however, values less than 0.10 are also
accepted (Kline, 1998).

Unlike the other fit indices, RMSEA is based

on the analysis of residuals, where smaller
values indicate a better fit to the data. It has
the advantage of going beyond point estimates
to the provision of 90 % confidence intervals
(Kelloway, 1998). In particular, RMSEA is a
parsimony-adjusted index. Its value decreases
as there are more degrees of freedom (greater
parsimony) or a larger sample size, keeping
the others constant (Kline, 2011). Values less
than 0.10 indicate a good fit to the data and
values less than 0.05 indicate a very good fit
(Schumacker and Lomax, 2004).
In SEM, hypothesis testing consists of
confirming that a theoretical specified model
fits sample variance-covariance data, by
testing
the
significance
of
structural
coefficients or testing the equality of
coefficients between groups (Schumacker and
Lomax, 2004). The initial model represents the
null hypothesis (Ho) and the final model

FADLELMULA

Table 2.Descriptions of Studies Included in the Meta-Analysis.

Study
Akn

Year
2008

Type of Journal
SSCI index

N
646

Sample
Undergraduate Students

Topic Area
Educational Sciences

Akn

2009

SSCI index

755

Undergraduate Students

Educational Psychology

Arcak

2009

SSCI index

695

Undergraduate Students

Educational Psychology

Aslan and Gven

2010

SSCI index

642

Undergraduate Students

Educational Psychology

etin, Gndz and

Akn
Ceylan and
Berberolu
Demir

2008

National index

357

Police Chief Candidates

Educational Psychology

2007

SSCI index

7841

Elementary School Students

2008

SSCI index

218

Teachers

Science and Technology

Education
Elementary Education

Ha laman and A kar

2007

SSCI index

730

Undergraduate Students

Technology Education

im ek,Glba and
Noyan
Uzun and retmen

2009

SSCI index

17647

Undergraduate Students

Educational Sciences

2010

SSCI index

7841

Elementary School Students

Uzun, Gelbal, and

retmen
Yavuz

2010

National index

7841

Elementary School Students

2009

SSCI index

310

High School Students

Science and Technology

Education
Science and Technology
Education
Secondary Education

(1996) method was used for the power

analyses.
represents the alternative hypothesis (Ha).
Each model produces a chi-square goodness
of fit value and the difference between these
values is used for testing significance.
Mathematically, this difference (denoted as
2
2
2
D = o - a with degrees of freedom dfd= df odfa) is tested for significance at a specified
2
alpha level where Ho is rejected if D exceeds
the critical chi-square value with df d degrees of
freedom (Schumacker and Lomax, p.113). In
this study, the alpha level was taken to be
0.05.
In this aspect, the power of SEM hypothesis
testing, which is the probability of rejecting Ho
when it is false, depends on the true
population model, significance level, degrees
of freedom, and sample size (Schumacker
and Lomax, 2004, p.113). Specially, for SEM
studies, MacCallum, Brown and Sugawara
(1996) proposed a method for calculating
power, in which power is considered in terms
of a null and alternative value of RMSEA
indices. This study reports a meta-analysis
performed to examine the power of research
studies conducted by using SEM technique,
and MacCallum, Browne and Sugawaras

METHOD
Literature Review Procedure
To select studies that included structural
equation modeling technique, an extensive
literature review was

39

conducted in Turkish Social Sciences

Database, available at the Turkish Academic
Network and Information Centre (ULAKBIM).
The journals included in the database provide
access to the national literature in the field of
social sciences, and they are evaluated by
ULAKBIM database committee according to
the Journal Evaluation Criteria conforming to
the international standards.
There were 176 journals included in this
database, which covered journals from 18
different subject matters. For the purpose of
this study, only journals related with education
field were considered. There were 34 journals
in this field. These journals were searched for
terms such as structural, structural model,

model, and structural equation modeling.

Through this process, more than 80 studies
were identified for possible consideration.
However, after a full text review, only 12
research studies, published from 2007 to
2010, were found suitable for the purpose of
this study, as they were the only studies
conducted with structural equation modeling
technique.
Characteristics of Selected Studies
The descriptive analyses of the selected
studies were summarized in Table 2.
Among these studies, two studies were
published in 2007, three were published in
2008, four were published in 2009, and three
were published in 2010. Moreover, 11

40

Educ. Res. J.

Table 3.Research Variables used for Calculating Power of SEM Studies

Study

df

Power

Study A

695

0.05

0.00

0.000

0.05

Study B

18

646

0.05

0.00

0.056

0.98

Study C

415

218

0.05

0.00

0.030

0.83

Study D

107

642

0.05

0.00

0.100

1.00

Study E

19

7841

0.05

0.00

0.082

1.00

Study F

18

755

0.05

0.00

0.029

0.49

Study G

7841

0.05

0.00

0.050

1.00

Study H

68

17647

0.05

0.00

0.057

0.92

Study I

844

730

0.05

0.00

0.044

1.00

Study J

310

0.05

0.00

0.090

0.72

Study K

284

7841

0.05

0.00

0.052

1.00

Study L

27

357

0.05

0.00

0.027

0.23

studies were published in Social Science Citation

Index (SSCI), and two were published only in
national
index.
Regarding
the
sample
characteristic, 6 studies were conducted with
undergraduate students, 1 was conducted with
high school students, 4 were conducted with
elementary school students, and 1 was
conducted with teachers. The sample size ranged
between 218 and 17,647. In addition, these
studies were conducted on many different subject
areas,
including
educational
sciences,
educational psychology, elementary education,
secondary education, and technology education.

Meta-Analytic Procedure
MacCallum, Brown and Sugawara (1996)
suggested method to calculate power in order to
measure the fit of structural equation models
based upon different fit indices, such as RMSEA
fit index. According to this method, in order to
perform a statistical power analysis, five factors
were needed to be taken into consideration.
These factors were degrees of freedom (df),
significance level (), sample size (N), the null
value of RMSEA ( 0), and the alternative value
of RMSEA ( a). Statistically in power analysis,
the difference between 0 and a refers

to the effect size, which is conceptualized as the

degree to which Ho is incorrect.
For this study, the null RMSEA value of each
study was taken as 0 = 0.00 for the test of exact
fit, as suggested by MacCallum et al. The other
research variables were obtained from the results
sections of each study or from the researchers
personally if the values were not

indicated in the original papers. Then, statistical

procedures were administered by Statistical
Analysis System (SAS) program and R routines,
which are free software environments for
statistical computing and graphics. In particular,
df, , N, 0, and a values were entered in the
SAS program. So, the program provided a code
for each set of variables. Then, this code was
inserted in R routines, which gave the exact value
of achieved power for each study.
RESULTS
Table 3 shows the research variables used for
power analysis of each study. The result of power
analysis revealed that 25% of the selected
studies achieved power less than 0.50.

Particularly, one study had a power of 0.05 (df =

1, = 0.05, N=695, and a= 0.00), one study
had a power of 0.23 (df = 27, = 0.05, N=357,
and
a = 0.027), and one study had a power of 0.49
(df = 18,
= 0.05, N=755, and a = 0.029). Among these
three studies, one had a very small degrees of
freedom (df = 1), one had a small sample size
(just around 300), and one had a very small
RMSEA value ( a = 0.029).
Further, among the other studies, one had a
power of 0.72 (df = 4, = 0.05, N=310, and a =
0.090), which was less than Cohens (1992)
conventions for an ideal power of 0.80.
Nevertheless, the other studies attained power
larger than 0.80, indicating good approximate fits.
Specially, five studies had full power of 1.00.
Among these five studies, three had large
amount of degrees of freedom (Study D, I, and
K), and two had large sample
FADLELMULA

size (Study E and G). During this meta-analysis,

none of the examined studies were found to
mention about the power issue of their structural
equation models.
DISCUSSION AND CONCLUSION
This study employed a meta-analytic procedure
to examine the statistical power of SEM studies
published in the Turkish Social Sciences
Database. After reviewing 34 journals in the
education field, 12 studies, published from 2007
to 2010, were found suitable for research
consideration. Power was calculated by SAS
program and R routines, using the indicated
degrees of freedom, significance level, sample
size,
and
root-mean-square
error
of
approximation indices. The results revealed that
one third of the selected studies achieved power
less than 0.80. Specially, three of them achieved
power less than 0.50.
Statistically in SEM studies, achieving a low
power implies that if the researchers model is
false, then the probability of detecting this
specification error is low (Kline, 2011). This
means that there is a high possibility that the
researcher accepts a false model as significant,
and s/he is unlikely to detect that that this model
is actually false. On the other hand, achieving a

41

indices, they achieved highly different levels of

power. In brief, power was low with small degrees
of freedom, sample size and RMSEA indices.
high power implies a higher probability of
detecting a reasonably correct model (Kline,
2011). Therefore, achieving low power can be
regarded as a highly serious problem as it
compromises the scientific value of the research
study. Especially, considering the fact that
rejecting a false null hypothesis with a power of
0.50 is the same as guessing the outcome of a
coin toss, the results of these studies can be
regarded as having very small practical
significance. In these cases, tossing a coin would
be more beneficial as it saves time and money
(Schmidt and Hunter, 1997).
Furthermore, the results of this study also
indicated how power of SEM studies varied as a
function of degrees of freedom, sample size, and
RMSEA indices. For instance, power was low
when there were small degrees of freedom, even
for a reasonably large sample size or RMSEA
value. Hence, for models with only one or two
degrees of freedom, sample sizes in thousands
were required for power to be greater than 0.80.
Like Study G, which had very small degrees of
freedom, sample size over 5000 was required for
achieving a high level of power. On the other
hand, for Study C, Study D, and Study I, which

had large degrees of freedom, powers were

reasonably high, even for sample sizes around
500.
Similarly, power was low when the difference
between the null and alternative value of RMSEA
was small, even for large sample sizes or degrees

of freedom. For instance, Study B and Study F

had similar amount of degrees of freedom and
sample size. However, due to the differences
between the null and alternative RMSEA

IMPLICATIONS
Power is a critical issue in designing and planning
research studies. Especially for SEM studies,
greater power implies a higher probability of
detecting a reasonably correct model (Kline,
2011). This is why; getting a significant result
from model testing is not enough to indicate that
the model fit is adequate. Indeed, before
tentatively concluding that the model is fit,
researchers need to pay special attention to
power level of their studies. In this study, power
was not issued in any of analyzed studies.
In the literature, there are a number of methods
for estimating power of structural equation
models, including those of Saris and Satorra
method (1993), MacCallum, Browne, and
Sugawara (1996), and Kim (2005). No matter
which method is implemented, researchers need
to estimate the power value of their study and to
ensure that the predetermined power is at least
0.80 (Cohen, 1992). For studies with power well
below this desired level, researchers need to
either change the conditions under which the
research would be conducted or postpone the
study until a reasonably high sample size is
reached.
REFERENCES
Akn A (2008). Self-compassion and achievement
goals: A structural equation modeling approach.
Eitim Ara trmalar, 31, 1-15.
Akn A (2009). Self-compassion and submissive
behavior. Eitim ve Bilim, 34(152), 138-147.
Arcak OT (2009). Psychiatric symptomatology as a
predictor of cyberbullying among university students.
Eitim Ara trmalar, 34, 167-184.
Aslan S, Gven M (2010). Balanma ve ki isel uyum
arasndaki ili kide ayr ma bireyle menin aracl.
Eitim ve Bilim, 35(157), 181-191.
Bollen KA, Long J S (Eds.). (1993). Testing structural
equation models. Newbury Park, CA: Sage.
Ceylan E, Berberolu G (2007). rencilerin fen ba
arsn aklayan etmenler: Bir modelleme al mas.
Eitim ve Bilim, 32(144): 36-48.
Cohen, J. (1992). Quantitative methods in psychology:
A power primer. Psychological Bulletin, 112(1): 155159.
etin B, Gndz HB, Akn A (2008). An investigation of
the
relationships
between
self-compassion,

motivation, and burnout with structural equation

modeling. Abant zzet Baysal niversitesi Eitim
Fakltesi Dergisi, 8(2): 39-45.
42

Demir K (2008). Transformational leadership and

collective efficacy: The moderating roles of
collaborative culture

Educ. Res. J.

and teachers self-efficacy. Eitim Ara trmalar, 33:

93-112.
Ha laman T, A kar P.(2007). Programlama dersi ile ilgili
zdzenleyici renme stratejileri ve ba ar
arasndaki ili kinin incelenmesi. Hacettepe
niversitesi Eitim Fakltesi Dergisi, 32: 110-122.
Jreskog KG, Srbom D (1993). LISREL 8: Structural
equation modeling with the SIMPLIS command
language. Chicago: Scientific Software International.
Karada E (2009). Ruhsal liderlik ve rgt kltr : Bir
yapsal e itlik modelleme al mas. Kuram ve
Uygulamada Eitim Bilimleri, 9(3): 1357-1405.
Kelloway EK (1998). Using LISREL for structural
equation modeling: A researchers guide. Thousand
Oaks: Sage Publications.
Kim KH (2005). The relation among fit indexes, power,
and sample size in structural equation modeling.
Structural Equation Modeling, 12: 368390.
Kline RB (2011). Principles and practice of structural
equation modeling (3rd ed.). New York: The Guilford
Press.
Kunnan AJ (1998). An introduction to structural
equation modeling for language assessment
research. Language Testing, 15(3): 295-332.
MacCallum RC, Browne MW, Sugawara HM (1996).
Power analysis and determination of sample size for
covariance structure modeling. Psychological
Methods, 1(2):130-149.
Pedhazur EJ, Pedhazur-Schmelkin LP (1991).
Measurement, design, and analysis: An integrated
approach. Hillsdale, NJ: Lawrence Earlbawm
Associate.
Raykov T, Marcoulides GA (2006). A first course in
structural equation modeling (2nd Edition). Lawrence
Erlbaum Associates, Inc. Publishers.
Sap M (2006). Basic psychological measurement,
research designs, and statistics without math.
Springfield, IL: Charles C. Thomas.
Saris WE, Satorra A (1993). Power evaluations in
structural equation models. In Bollen KA, LongJS
(Eds.), Testing structural equation models (pp. 181
204). Newbury Park, CA: Sage.

Schumacker RE, Lomax RG (2004). A beginners

guide to structural equation modeling. Mahwah NJ:
Lawrence Erlbaum.
Schmidt FL, Hunter JE (1997). Eight common but false
objections to the discontinuation of significance
testing in the analysis of research data. In Harlow
LL, Mulaik SA, JH Steiger (Eds.), What if there were
no significance tests? Mahwah, NJ: Erlbaum. pp.
3764.
im ek GlbaZ G, Noyan F (2009). The effect of
perceived instructional effectiveness on student
loyalty: A multilevel structural equation model.
Hacettepe niversitesi Eitim Fakltesi Dergisi,
36;109-118.
Uzun B, retmen T (2010). Fen ba ars ile ilgili baz
dei kenlerin TIMSS-R Trkiye rnekleminde
cinsiyete
gre
lme
dei
mezliinin
deerlendirilmesi. Eitim ve Bilim, 35(155), 26-35.
Uzun NB, Gelbal S, retmen T (2010). TIMSS-R fen
ba ars ve duyu sal zellikler arasndaki ili kinin
modellenmesi ve modelin cinsiyetler bakmndan kar
la trlmas. Kastamonu niversitesi Kastamonu
Eitim Fakltesi Dergisi, 18(2): 531-544.
Yavuz M (2009). Ortaretim kurumlar renci seme
ve yerle tirme snavnda rencilerin Matematik-Fen (
MF ) puanlarn etkiledii d nlen baz faktrlerin
yapsal e itlik modeli ile incelenmesi. Kuram ve
Uygulamada Eitim Bilimleri, 9(3): 1543-1572.