Causal Comparative Research Designs

The Study of Cause and Effect Relationships

As we explained at the start of the chapter, some quantitative research designs have the purpose of
explaining educational phenomena through the study of cause and effect relationships. In these designs, the
presumed cause is called the independent variable, and the presumed effect is called the dependent variable.
For example, suppose we hypothesize that the introduction of state-mandated testing of all students will have
a negative effect on teachers' morale. In this hypothesis, state-mandated testing is the independent variable
(i.e., the presumed cause of a drop in teachers' morale), and teacher morale is the dependent variable (i.e., the
presumed effect of the introduction of state mandated testing).

In another hypothesis, this dependent variable (teacher morale) might become the independent variable. For
example, we might hypothesize that teacher morale affects their rate of absenteeism from work. In this
hypothesis, teacher morale is the independent variable, and absenteeism rate is the dependent variable.

In the rest of this chapter and the next, we consider research designs in which the researcher does not
manipulate the independent variable in order to observe its effect on the dependent variable. These research
designs do not permit strong conciusions about cause and effect, but are useful for initial exploratory
investigations or in situations where it is impossible to manipulate the independent variable. For example,
researchers might want to know whether teacher morale affects absenteeism. It would be virtually
impossible for them to conduct a study in which they create work conditions that promote good morale
among some teachers and poor morale among other teachers. Instead, they are limited to observing naturally
occurring variations in teacher morale. These variations can be measured and related to naturally occurring
variations in teacher absenteeism rates.

Research designs that rely on observation of relationships between naturally occurring variations in the
presumed independent and dependent variables sometimes are called ex post facto research.l5 (Ex postfacto
is a Latin phrase meaning “operating retroactively") In contrast, experiments involve actual manipulation of
the independent variable by the researcher.

Causal comparative research is a type of nonexperimental investigation in which researchers seek to identify
cause-and-effect relationships by forming groups of individuals in whom the independent variable is present
or absent-or present at several levels-and then determining whether the groups differ on the dependent
variable. The critical feature of causal comparative research is that the independent variable is measured in
the form of categories. The categories can form a nominal scale (e.g., male versus female; American versus
Asian versus European citizenship) or ordinal scale (e.g., nonemployed versus employed part-time versus
employed full-time). This approach to measuring the independent variable lends itself to particular statistical
methods for analyzing the resulting data, which we describe in this chapter.

It is important to realize that an independent variable measured in categorical form might also be measurable
as an interval or ratio scale. For example, consider the variable of employment. In designing a research
study, we can choose to measure employment on a simple ordinal scale: the research participants are either
employed or not employed. However, it is possible to measure employment more precisely as a ratio scale
consisting of amount of hours of weekly employment-for example, 0 hours, 1.5 hours, 20 hours, 25 hours, 40
hours In this case, a correlational research design (see Chapter 11) is more appropriate for organizing the
independent and dependent variables and analyzing the resulting data.
In fact, any causal-comparative research design can be reconceptualized as a correlational research design by
changing how the variables are measured or analyzed. or both. However, researchers sometimes prefer to
use a causal-comparative design for two reasons: forming groups to measure the independent variable often
is more consistent with how practitioners and other education stakeholders think about the world; and the
statistical results typically are easier to comprehend and interpret.
The design of causal-comparative research studies can vary. The researcher can plan to include one
independent variable or several in the design. Similarly, the researcher can choose to include one dependent
variable or several. Also, the choice of statistics will vary depending on characteristics of the research data.
As you read about causal-comparative designs in this chapter, you will be laying the foundation for your
study of experimental research designs in Chapters 13 and 14. This is because both casual comparative
research and experimental research involve the investigation of independent and dependent variables. You
will find that experimental designs look similar to causal-comparative designs in terms of how the variables
are organized and displayed. Also, you will find that some of the same statistical methods are used to
analyze the resulting data.
Example of a Causal-Comparative Research Study
Leslie Vandevoort, Audrey Amrein-Beardsley, and David Berliner used a causal-comparative research
design to investigate whether earning a certificate from the National Board for Professional Teaching
Standards has a subsequent positive effect on students' academic achievement.”
The National Board was established in 1987 to determine standards of superior teaching and to develop a
method of evaluating and certifying whether a teacher meets those standards. The evaluation process is time
consuming and costs thousands of dollars, but teachers who meet the standards become board-certified,
which is meant to confer prestige on them and assure school administrators and the community that they are
highly effective in promoting students' learning. However, as Vandervoort and her colleagues note, there is
little research to test the assumption that board certified teachers actually bring about higher levels of student
learning than teachers who are not board certified. The purpose of the study was to empirically investigate
this assumption.
As indicated above, causal-comparative research is non experimental. In an experiment, Vandevoort and
colleagues would have formed a sample of teachers and randomly assigned half of them to prepare for, and
go through, the National Board's evaluation process and the other half to continue with their usual work
patterns. However, it would be virtually impossible in the current schooling system to require any teacher to
spend months preparing for the National Board evaluation and then submit to the demanding evaluation
process, especially when there is a substantial risk of failure. Therefore, Vandervoctt and her colleagues
needed to look for natural, preexisting variations in the independent variable (holding or not holding
National Board certification). For this reason, the researchers characterized their study as "an ex-post facto
causal-comparative research design.”
Vandevoort and her colleagues were able to locate 35 board-certified teachers in early childhood or middle
Childhood education in Arizona, the state where the study was conducted. The comparison group consisted
of nonboard-certifted teachers working at the same grade levels in the same districts as the board-certified
teachers.
The dependent variable in the research design was students’ achievement gain over a period of 1 year on the
SAT-9, a norm-referenced achievement test that measures skills in reading, math, and language. Gain was
assessed by determining each student's SAT-9 scores in, let's say, 2002 and then again in 2003. Students
were assigned to the board-certified condition if they had a nonboard-certified teacher in 2002 and a board
certified teacher in 2003. Students in the non board-certified condition had a non board ertified teacher both
years. (In fact, the researchers created built-in replications of their research design by studying three school
years, including 2002-2003.)
Because students were not randomly assigned to have, or not have, a board-certified teacher in 2003, it is
possible that the two groups of students differed in initial academic achievement as measured by the SAT-9
in 2002. If so, this could compromise the research design: For example, if students of board-certified
teachers had lower achievement than the comparison group in 2002, they might make greater achievement
gains over the 1-year period simply because it is sometimes easier to go from a very low score tu a moderate
score than from a moderate score to a higher score (see Chapter 7). Thus, the greater gains of students
having boaid-certified teachers could be attributed to their initial achievement level, not their exposure to
superior teaching. To rule out this possibility, the researchers adjusted the gain scores using a statistical
procedure known as analysis of covariance. which is described later in the chapter. In effect, analysis of
covariance brings both groups of students to the same initial level of achievement on the SAT-9 so that any
observed differences in gain scores can be attributed to the independent variable (National Board
certification) rather than differences in initial achievement.
Table 10.1 shows the mean and standard deviation of adjusted gain scores for students at four different grade
levels taught by National Board-certified teachers and non board certified teachers. Separate adjusted gain
scores are shown for students’ performance on the reading, math, and language sections of the SAT-9. The
table shows whether the difference between the mean scores of the two groups is statistically significant at
the p < 05 level. In addition, the researchers present the effect size for each comparison. which is a measure
of the practical significance of each observed difference between students of board-certified and non board
certified teachers. (See Chapter 5 for an explanation of effect sizes and tests of statistical significance.)
For example, we find in Table 10.1 that third-graders taught by board-certified teachers made an average
adjusted gain of 38.6 points on the SAT-9 reading test, whereas third graders taught by non board certified
teachers made a smaller average adjusted gain of 31.2 points. The difference between the two mean gains
(38.6 and 31.2) was statistically significant, meaning that it is unlikely that the null hypothesis of no
difference between the populations that these two groups represent (students of board certitied teachers and
students of non board certihed teachers) is correct. In 10 of the 12 comparisons, studems tanght by board-
certitied teachers made greater gains than those taught by non board certitied teachers. Also. the magnitude
of the effect-size difference between the two adjusted mean gains (.335) is substantial. As Vandevoort and
her colleagues state:
In 75.0% (9/12) of the total cases, [335 were larger for students of NBCTs [National Boardcertified teachers]
than for students of non-NBCls. The ES in favor of the students of the NBCls averaged .225 in reading, about a
two and a quarter months’ advantage; .065 in mathematics, over a half month’s advantage; and .047 in
language, representing almost a half month's advantage.

A disadvantage of causal-comparative research designs is that inferences about causality on the basis of the
collected data are necessarily tentative. Therefore, we must be cautious about concluding from these results
that the process of achieving National Board certification will make teachers more effective in promoting
students' academic achievement. Alternative interpretations of the results are possible. For example, it might
be that effective teachers (defined as teachers whose students make good gains on achievement tests) are
more likely to seek and earn National Board certification. If so, we can say that a third variable C, teachers’
initial teaching effectiveness, has a positive causal influence on variable A, the desire to seek and earn a
National Board certificate, and variable B, their students' score gains on achievement tests. In this
interpretation, which cannot be ruled out by the causal-comparative study described here, variable A does
not have a causal influence on variable B. It only seems that way because a third variable, teachers' initial
teaching effectiveness, influences the other two variables.
Another interpretation is that there is an actual causal relationship between National Board certification and
student achievement gains, but the direction of causality is the reverse of what was hypothesized. How can
this happen? It might be that board certified teachers are assigned classes that have higher-achieving students
initially. These students might make substantial achievement gains over the school year, thus making their
teachers look good. Nonboard-certitied teachers might be assigned classes with typical or lowerachieving
students who would make lesser achievement gains. Vandevoort and her colleagues, to their credit,
attempted to equalize the initial achievement level of Students in both groups by a statistical technique,
analysis of covariance. However, analysis of co variance does not always work as intended. In the present
study, for example, it might equate students in the two groups on academic-achievement performance, but
not on other critical variables.
The safest interpretation of this causal-comparative study is that it provides evidence suggesting that the
process of preparing for and passing the assessments for National Board certification improves teachers’
ability to help students learn. This tenttative evidence is valuable, because it makes the case for investing in
studies involving an expert mental design (see Chapters 12 and 13), which typically are expensive, but that
enable more definitive conclusions about the causal effect of National Board certification on student learning
in school.