See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/258723133

Value-added measures in education: What every educator

needs to know

Article  in  Science Education · March 2012

DOI: 10.1002/sce.20484

CITATIONS READS

5 612

1 author:

Gavin W. Fulmer
University of Iowa
40 PUBLICATIONS   197 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Alignment for Education Standards, Assessments, Curriculum, and Teaching View project

Moving beyond pedagogy: Developing elementary teachers' adaptive expertise in using the epistemic complexity of science View
project

All content following this page was uploaded by Gavin W. Fulmer on 20 December 2017.

The user has requested enhancement of the downloaded file.

Science THE BOOKS
Education

Value-Added Measures in Education: What Every Educator Needs to Know, by

Douglas N. Harris. Harvard Education Press, Cambridge, MA, USA, 2011. ix + 276
pp. ISBN 978-1-6125-0000-3.

In Value-Added Measures in Education: What Every Educator Needs to Know, Douglas

Harris seeks to explain value-added modeling (VAM) so that readers will understand the
logic, strengths, and limitations of this method for evaluating schools and teachers. A sec-
ondary goal of the book is to convince readers that, when used appropriately, VAM can be a
useful tool in school accountability efforts. Its target audience is teachers, school adminis-
trators, and other interested readers with little experience in statistics or educational policy.
The book begins with an explication of accountability efforts and what Harris calls
the cardinal rule of accountability: “Hold people accountable for what they can control”
(p. 4). This concept serves as the major theme for the rest of the text. Harris argues that
VAM comes closer to meeting this rule than current approaches, such as adequate yearly
progress (AYP) based on cross-sectional test data. That is because cross-sectional tests (e.g.,
fourth-grade statewide science tests) compare the students in the current year to students
from prior years. The cross-sectional approach cannot separate the effects of teachers and
schools from other, confounding variables over which schools and teachers have little or
no control–such as student socioeconomic status (SES), district resources, class size, or
students’ prior knowledge.
The next section of the book describes VAM and its recent applications. Harris devotes
a chapter in this section to concepts of systematic and random error to ground the later
presentation of the strengths and limitations of VAM. The purpose of this section is to
compare VAM and cross-sectional methods on how well they can limit such statistical
errors. As Harris shows, VAM determines students’ growth in achievement by comparing
each student’s current-year score with his or her own prior-year score, rather than comparing
current-year students with a different group of prior-year students. VAM then determines
how students’ growth in achievement in a class or a school differs from the average
growth in other classes or in other schools. The “average” can also be adjusted based on
the estimates of other variables in the model, such as student SES, class size, or prior
knowledge. This allows teachers or schools to be identified that perform better at helping
students achieve above the “average” for students of similar background in schools with
similar circumstances. In this way, VAM can yield estimates of the effects of schools or
teachers that account for predictors outside the control of the teacher and school. Therefore,
VAM has much greater potential to distinguish teacher or school effectiveness from other
predictors that educators cannot control.
The remainder of this section of the book is devoted to concerns with VAM. Despite the
exciting potential for VAM, there are also serious concerns and Harris addresses several.


C 2012 Wiley Periodicals, Inc.
 BOOK REVIEWS 365

Two of these issues were most salient in my reading of the book. The first major concern
is about the assessments upon which VAM is built. VAM always requires at least two test
scores for each student. VAM estimates of students’ learning will not be valid if the two
tests do not measure the same constructs. In addition, the inferences from the VAM will
not be valid if the test does not measure relevant knowledge and skills that align with
the curriculum. In cases where the assessments are not satisfactory, the results of VAM
will be equally problematic, as highlighted by the author’s call to the “garbage in garbage
out” principle (p. 31). Furthermore, some outcomes that may be important for school and
for citizenship may not be easily measured by current standardized assessments, such as
citizenship behaviors, creativity, or socioscientific reasoning.
A second issue that Harris identifies with VAM is that it does not provide statistically reli-
able distinctions among schools or teachers near the average. This is because the confidence
intervals for VAM estimates can be large, especially near the middle of the distribution.
Harris argues that VAM still makes it possible to distinguish very low-performing from
very high-performing teachers or schools. However, attempts to distinguish either low or
high performers from the middle performers have been much more problematic because of
the standard errors. As such, decisions about what proportion of teachers to consider “low”
or “high” performers—whether 5% or 20%—are mostly arbitrary.
The book’s final section addresses appropriate uses of VAM. Harris points out that
many of the issues identified are not unique to VAM. In terms of assessment, current
cross-sectional AYP decisions also rely on the standardized tests that are needed for VAM.
Arbitrary cutoffs also exist in decisions about students’ proficiency levels on state standard-
ized tests (cf. National Center for Education Statistics, 2007). Therefore, VAM can provide
somewhat better data for making decisions about school effectiveness than approaches based
on AYP, but VAM cannot resolve the key debates in assessment and accountability policy.
Building on these limitations, Harris suggests that VAM estimates about teachers and
schools should be just one of a suite of tools used for evaluating teachers or schools. He also
argues against the use of VAM to estimate individual teacher’s performance and in favor
of VAM for schools or teams of teachers (such as grade bands or other structures) because
this reduces the threat of random errors. Harris’ argument against evaluating individual
teachers and for evaluating schools is clear and builds soundly on the logic of VAM and
the limitations presented in the book. The argument for evaluating teams of teachers is
less well articulated, which could be strengthened by a more thorough discussion of the
literature on teams in organizational psychology or in educational policy. Furthermore, he
does not address satisfactorily the challenges of calculating value added for a team and
how this would translate into actionable information for individual teachers to adapt their
instruction or for administrators to make personnel decisions.
The author is clearly making an effort to connect directly with educators and school
administrators. The book includes samples from actual data and figures from school districts
with which he has worked (presented with pseudonyms). The author also presents a vignette
about a teacher, who inspired him, which served as a topic for comparing how VAM might
or might not be able to detect her value as a teacher.
Stylistically, the book is written in the first person, using clear prose, with very little
jargon or academic language. The presentation should be accessible for readers who are
unfamiliar with statistics. Unfortunately, the current printing is plagued by distracting typos
and other editorial errors. Researchers interested in pursuing scholarship in valued-added
modeling studies should note that the book is not intended to be a primer on VAM. It
does not demonstrate how to estimate and interpret such models in practice. For a more
academic introduction to VAM, I recommend the book by McCaffrey, Lockwood, Koretz,
and Hamilton (2003) or the volume edited by Lissitz (2006). It appears that VAM and similar
Science Education, Vol. 96, No. 2, pp. 364–366 (2012)
366 BOOK REVIEWS

approaches to school and teacher accountability may play an increasing role in educational
policy over the coming years. This book will be a welcome volume for educators and
administrators who want to understand VAM’s potential promise and pitfalls.

REFERENCES
Lissitz, R. W. (Ed.). (2006). Longitudinal and value added models of student performance. Maple Grove, MN:
JAM Press.
McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003). Evaluating value-added models for
teacher accountability. Santa Monica, CA: Rand Corporation.
National Center for Education Statistics. (2007). Mapping 2005 state proficiency standards onto the NAEP scales:
Research and development report. Document number NCES 2007-482. Washington, DC: U.S. Department of
Education.

GAVIN W. FULMER
Arlington, VA 22230 USA
DOI 10.1002/sce.20484
Published online 9 February 2012 in Wiley Online Library (wileyonlinelibrary.com).

Science Education, Vol. 96, No. 2, pp. 364–366 (2012)

View publication stats