Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
DETAILS
CONTRIBUTORS
GET THIS BOOK Alexandra Beatty, Rapporteur; Board on Testing and Assessment;
Division of Behavioral and Social Sciences and Education; National
Academies of Sciences, Engineering, and Medicine
FIND RELATED TITLES
Visit the National Academies Press at NAP.edu and login or register to get:
Distribution, posting, or copying of this PDF is strictly prohibited without written permission of the National Academies Press.
(Request Permission) Unless otherwise indicated, all materials in this PDF are copyrighted by the National Academy of Sciences.
This activity was supported by Grant No. 7717 from the S.D. Bechtel, Jr. Founda-
tion. Any opinions, findings, conclusions, or recommendations expressed in this
publication do not necessarily reflect the views of any organization or agency that
provided support for the project.
Additional copies of this publication are available for sale from the National
Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800)
624-6242 or (202) 334-3313; http://www.nap.edu/.
The National Academy of Engineering was established in 1964 under the c harter
of the National Academy of Sciences to bring the practices of engineering to
advising the nation. Members are elected by their peers for extraordinary con-
tributions to engineering. Dr. C. D. Mote, Jr., is president.
For information about other products and activities of the National Academies,
please visit nationalacademies.org/whatwedo.
vi
Acknowledgments
T
his Proceedings of a Workshop has been reviewed in draft form
by individuals chosen for their diverse perspectives and technical
expertise. The purpose of this independent review is to provide
candid and critical comments that will assist the institution in making its
published Proceedings of a Workshop as sound as possible and to ensure
that the Proceedings of a Workshop meets institutional standards for clarity,
objectivity, and responsiveness to the charge. The review comments and
draft manuscript remain confidential to protect the integrity of the process.
We thank the following individuals for their review of this Proceed-
ings of a Workshop: Catherine Bradshaw, Research and Faculty Develop-
ment, Curry School of Education, University of Virginia; Ellen S. Gannett,
National Institute on Out-of-School Time, Wellesley Centers for Women,
Wellesley College; and Patrick H. Tolan, Youth-Nex, Center to Promote
Effective Youth Development, Curry School of Education, University of
Virginia.
Although the reviewers listed above have provided many constructive
comments and suggestions, they did not see the final draft of the Proceed-
ings of a Workshop before its release. The review of this Proceedings of a
Workshop was overseen by Christopher Cross, Cross & Joftus, Danville,
California.
He was responsible for making certain that an independent examina-
tion of this Proceedings of a Workshop was carried out in accordance
with institutional procedures and that all review comments were carefully
considered. Responsibility for the final content of this Proceedings of a
Workshop rests entirely with the rapporteur and the institution.
vii
Contents
1 Introduction 1
6 Measuring Character 63
Methodological Issues, 63
Perspectives on Measurement Challenges, 71
ix
x CONTENTS
7 Workshop Themes 77
Participant Perspectives on Key Questions, 77
Closing Thoughts, 79
References 81
Appendixes
A Workshop Agenda 85
B Participant List 91
C Biographical Sketches of Committee Members and Presenters 95
D Worksheet for Breakout Sessions 107
Introduction
T
he development of character is a valued objective for many kinds of
educational programs that take place both in and outside of school.
Educators, parents, and others create and support structured pro-
grams and lessons that engage students in academics, sports, service, and
other activities with the aim of developing or strengthening positive behav-
iors, attitudes, values, and attributes. Programs that pursue this kind of
learning may describe what they do as character education; positive youth
development; or the development of social and emotional learning, inter
personal and intrapersonal competencies, or noncognitive skills. These terms
are not interchangeable, but there is overlap among them: They encompass
a range of skills and attributes that students need to flourish in school, the
workplace, and their personal lives, such as the capacity to manage their
emotions, set and achieve positive goals, feel and show empathy, maintain
positive relationships, and make sound decisions. This loosely defined set of
skills and attributes is referred to in this document as character.
Educators and administrators who develop and run programs that seek
to develop character recognize that the established approaches for doing so
have much in common, and they are eager to learn about promising prac-
tices used in other settings, evidence of effectiveness, and ways to measure
the effectiveness of their own approaches. The available research has been
sparse and often focused only on one kind of character or development, but
recent work has helped to identify commonalities in the literature that can
advance understanding of how character might be defined and developed
and how outcomes might be measured.
With the support of the S.D. Bechtel, Jr. Foundation, the National
BOX 1-1
Committee Statement of Task
INTRODUCTION 3
BOX 1-2
Clark McKown, Promises and Perils of Assessing Character and Social and
Emotional Learning
The committee commissioned eight papers (see Box 1-2) and planned
sessions that allowed participants ample time to engage with the authors
and one another, and to consider ways the material presented could apply
in their own work.2 Structured breakout sessions allowed participants to
2The workshop agenda, a list of participants, and brief biographical sketches for the com-
mittee members and presenters can be found in Appendixes A, B, and C. The commissioned
papers and an archived video of all sessions can be found at the project website, http://sites.
nationalacademies.org/DBASSE/BOTA/DBASSE_171735 [September 2016].
A
lthough there are many definitions of character, adults can best help
young people develop it if they have a clear idea of what it is and
what it is not, moderator Richard Lerner commented in introducing
a discussion of the nature of character. While character is a word with
wide application in everyday language, he emphasized, efforts to develop it
in young people are focused on something that is not a trait but a devel
opmental phenomenon. A persons character is not fixed by genes,
he added, but is one outcome of his or her context and experiences. The
session began with a synthesis of scholarship on moral reasoning and char-
acter presented by Larry Nucci of the University of California, B erkeley,
who also proposed a model for understanding the nature of character.
Follow-up discussion was anchored by reflections from Robert McGrath of
Fairleigh Dickinson University, Kristina Schmid Callina of Tufts University,
and Carola Surez-Orozco of the University of California, Los Angeles.
1Quoted from his speech delivered August 28, 1963; available at https://kinginstitute.
stanford.edu/king-papers/documents/i-have-dream-address-delivered-march-washington-jobs-
and-freedom [November, 2016].
2See https://www.templeton.org/what-we-fund/core-funding-areas/character-virtue-development
[November 2016].
WHAT IS CHARACTER? 7
principles of justice and fairness. This idea has also been criticized, Nucci
noted, but the idea that human beings share a concern for justice and the
welfare of others is still widely accepted. Nucci agreed that the concern is
universal and commented that any meaningful notion of character has to
place morality at the center. Many of the character attributes that have
been highlighted in recent research, he explained, such as grit or social
and emotional intelligence, might be deployed for either positive or nega-
tive goals: These traits contributed to the practical success of the Nazis or
members of ISIS, for example. Such traits may be important in the pursuit
of personal and social benefits, such as success in school or career, but do
not in themselves lead to moral or immoral actions. Performance itself, he
added, is not a sufficient indicator of a persons character.
Some researchers who study moral education have moved away from the
term character, Nucci explained, and use the terms moral self or moral
identity instead. These terms refer not to a set of traits, he explained, but
a system that is a component of an individuals overall sense of self. This
overall sense of self includes the individuals sense of agency, unique personal
identity, gender and ethnic identity, and sense of him or herself as a produc-
tive member of family and society. The character system that operates within
that larger system, he explained, consists of the interactions among elements
of the self, such as self-regulation or executive control.
Many scholars of moral education have studied well-known individuals
who seem to exemplify moral behavior, on the theory that morality is more
central to their sense of self than it is for others. It turns out, however, that
a close examination of such individuals lives shows that these individuals
are not very different from other people, Nucci explained. The behavior
moral cognition,
emotional development or moral mental health,
WHAT IS CHARACTER? 9
Self System
performance, and
moral (critical) social engagement.
BOX 2-1
Childs Reasoning about a Moral Issue
Did you see what happened? Yes. They were playing and John hit him too hard.
Is that something you are supposed to do or not supposed to do? Not so hard to hurt.
What if there were no rule about hitting hard, would it be all right to do then? No.
BOX 2-2
Childs Reasoning about a Conventional Issue
Did you see what just happened? Yes. They were noisy.
Is that something you are supposed to do or not supposed to do? Not do.
WHAT IS CHARACTER? 11
different faith traditions about their views of moral issues. For example,
they asked children from an Amish community in Indiana whether it would
be wrong or right to remove a rule prohibiting particular actions. Nearly
all the children interviewed reported that it would be wrong to remove
rules against actions with a clear moral component, such as stealing or
hitting, but significantly fewer reported that it would be wrong to change
rules regarding nonmoral issues, such as worshipping on a particular day
of the week.
When asked for their reasons, the children were most likely to cite
Gods law, Nucci explained, as the reason it would be wrong to change
rules about moral issues, but they also cited other reasons, such as the
welfare of others and fairness. The children were also asked to consider
whether their answers would be different if God had not given a rule
for either the moral or nonmoral issues. Virtually none of the children
responded that the nonmoral rules should be changed, but significant pro-
portions still argued that it would be wrong to change moral rules, such as
those against stealing or hitting, even if God had not said anything about
these actions. In those cases, the childrens judgments were focused on the
impacts of the actions on other people.
Judgments about right and wrong, Nucci concluded, reflect an indi-
viduals capacity to negotiate three domains: the moral, the conventional,
and the personal. Each domain puts varying demands on an individual
over time, just as an individuals priorities shift over time. An individuals
priorities are also affected by the facts and information he or she has and
the assumptions he or she makes about that information. For example, a
person who assumes a human egg is a person may accordingly view abor-
tion as an immoral act of murder, while a person who does not view the egg
as a person will disagree. Science, religious belief, and cultural traditions
all contribute to such assumptions, Nucci added, and thus critical thinking
also plays a part in morality.
Individuals develop in each of the attributes that contribute to moral
judgment, Nucci went on, but there are no defined stages of development in
peoples capacity to coordinate competing considerations in complex social
situations. Nor is there an end point, he addeda stage at which people
have the wisdom to apply moral principles in all contexts, regardless of
competing nonmoral considerations. In other words, Nucci noted, moral
judgments are inexorably bound up in context, and this makes the assess-
ment of moral growth and the identification of character more challenging.
Component 3: Performance
Character entails not only the capacity to recognize the right thing to
do but also the propensity to act on that judgment, Nucci explained.
Doing the right thing often comes at a cost, he went on, and in some cases
the cost may be so high that it is completely rational to prioritize self-
interest over the morally right thing to do. Nucci cited as an example
cases in which child soldiers are ordered to take immoral actions and face
dire consequences for refusing to obey. Acting on moral judgments is not
simply a matter of willpower or motivation, Nucci argued. Behaving in
a moral fashion even when doing so competes with other goals requires a
capacity for self-regulation and executive function (the processes that allow
people to control their own behavior). Recent work on grit, defined as the
capacity to both feel passion for a long-term goal and persevere in pursuing
it (Duckworth, 2016), suggests that it may be an important element of the
capacity to act morally. For example, grit might help to explain individuals
commitment to addressing injustices despite extreme challenges, but does
not in itself account for the moral thinking that led to that commitment,
he noted.
WHAT IS CHARACTER? 13
3Nucci had included this point in an earlier draft of his paper, which McGrath used in plan-
WHAT IS CHARACTER? 15
WHAT IS CHARACTER? 17
4The model is based on the book Character Strengths and Virtues (Peterson and Seligman,
McGrath believes there are strong reasons for viewing these three
strengths as essential or universal virtues, but he also acknowledged the
importance of social context, which continually redefines what is meant
by these essential virtues. For example, both the invention of the printing
press and the development of the Internet brought radically new contexts
for human inquisitiveness. McGrath suggested that these three strengths are
necessary elements of character but not sufficient. Virtues and strengths
interact, he said, noting that this idea goes back at least to Aristotle, who
argued that people who have the wisdom to develop one virtue logically
therefore possess other virtues (this is the thesis of reciprocity of virtue).
Character education that focuses on only one virtue or strength, however
important, is insufficient, in his view. In teaching people about virtue, he
concluded, we need to recognize that people have personal, interpersonal,
and cultural values and that together these three contribute to a life well
lived. In his view, effective character development must include teaching
young people to be caring without the expectation of benefit, to be ques-
tioning without a crisis, and to be disciplined without structure.
WHAT IS CHARACTER? 19
must account in some way for the interactions between the person and the
context, she added.
Across time and place, good character reflects the coherence of an indi-
viduals behaviorhow reliably he or she displays the right virtue, in the
right amount, at the right time, in Aristotles words. The attributes needed
might vary according to circumstances, Schmid Callina noted, so what is
critical is coherence, not consistency. Focusing on the idea that character
traits are fixed, or on the importance of single traits, will likely prevent one
from seeing the coherence in an individuals actions.
Schmid Callina described a study of character and leadership develop-
ment among cadets at the U.S. Military Academy at West Point, called
Project Arete, to illustrate how ideas about the importance of context and
coherence can be used to evaluate character development. The most com-
mon method of evaluating character development curricula, she noted, has
been to assess particular strengths in young people and then compare their
scores from before and after completing the program. This approach often
fails to integrate the idea of coherence.
Specifically, such evaluations examine average group differences on
particular character attributes, rather than the individuals relationship with
the developmental system, Schmid Callina explained. They do not examine
the individuals developmental pathways or interactions with his or her con-
text, which would provide insight about character development. To explore
character development among West Point cadets, she explained, one might
examine the attributes they bring that make them more likely to develop
as leaders of character, or what about the West Point experience influences
their character development. These are important, she noted, but to focus
just on individual attributes, or on characteristics of the program, as com-
ponents is to overlook their interactions. More useful, she explained, is to
examine which features of the program promote which attributes among
cadets at a particular time and place.
One challenge in examining individual pathways to development, she
explained, is that standard approaches to statistical analysis in the social
sciences tend to focus on differences between people rather than changes
within individuals.6 Integration of multiple methods of qualitative and
quantitative research is needed, Schmid Callina noted, to effectively ana-
lyze changes within individuals and then aggregate such findings to yield
broader conclusions about character development. Person-centered analy-
sis is critical, she argued for evaluating programs such as that at West
Point. There are as many different experiences at West Point as there are
cadets, she added, to highlight the importance of using new approaches to
understand them. This is a lively area of research, she noted in closing, as
many new tools are being developed. Such research is expensive and time
consuming, but in her view very promising.
6Schmid Callina recommended The End of Average: How We Succeed in a World That
Values Sameness by Todd Rose (2015) for a detailed discussion of these issues.
WHAT IS CHARACTER? 21
articulated the problem and identified 152 definitions of the term. They syn-
thesized from these many definitions five essentials: values, beliefs, r ituals,
symbols, and practices. Human values are at the center, Surez-Orozco
explained, as Figure 2-4 illustrates, and each of these elements is expressed
through humans day-to-day practices.
Anthropologists pay close attention to cultural practicesthe routines,
activities, and other things that people doSurez-Orozco commented.
These include
language use;
kinship systems;
religious and ritual practices;
economic models;
power structures and hierarchies;
gender expectations;
cultural socialization (child rearing);
dress; and
food.
WHAT IS CHARACTER? 23
viduals and also influence U.S. society, she added (Lerner, Dowling, and
Anderson, 2003; Stepick, Stepick, and Labissiere, 2008). Surez-Orozco
and her colleagues explored the degree to which these young people follow
patterns of civic engagement typical in the broader population, and also
sought to understand the values and motivations that drive them to engage
civically.
They recruited a sample of 58 young adults of Dominican, Mexican,
Guatemalan, and Salvadoran origin who are first- or second-generation
immigrants living in cities in the Northeast. The participants were asked to
complete questionnaires and to participate in a Q-sort exercise, in which
they ranked their views and values. Figure 2-7 summarizes the participants
responses to the task of picking four values (from among 20) they saw as
most associated with the United States, their or their parents country of
origin, and themselves.
Independence was given a high priority across all three groups, she
noted, but some other values were specifically associated with only one or
two. For example, pursuit of wealth, freedom, and opportunity were associ-
ated with the people of the United States generally, but not the participants
countries of origin or themselves. The young adults associated religion only
with their countries of origin, and highlighted family obligations, helping
and serving others, and respect, along with independence, as important to
themselves.
WHAT IS CHARACTER? 25
DISCUSSION
Discussion focused on the tension between the degree of emphasis
placed on the individual versus the cultural context that was evident in the
presentations, as one participant put it. The participant pointed out that,
although people are clearly influenced by their cultural environment, there
is a biological substratum and a genetic influence. People do have an un-
changeable core of traits that would persist even if they were transplanted
to an entirely new cultural context, he argued.
Nucci responded by acknowledging that each individual does have a
unique identity that is stable, but that reducing character to a set of immu-
table traits is misguided. Martin Luther King, Jr., he reminded the group,
was both an esteemed moral leader and a philanderer, who presumably
was dishonest to family members and others to hide his infidelity. Thinking
about character in terms of coherenceto attempt to understand how an
individuals internal motivation and reasoning interact with the circum-
stances and external influencesis more useful for making sense of this
type of paradox than looking for consistent traits in the individual, Nucci
suggested. If consistency is the standard, we all fail, he pointed out.
Others turned the discussion to practical implications for character
education. One suggested that focusing on the commitment to be a better
7This legislation was first proposed in the United States Senate in 2001 but has not yet
passed.
person is a way around the debate over the respective influences of indi-
vidual attributes and cultural influences. Programs that help young people
develop character could usefully be viewed as opportunities for young
people to practice acting as people of character, making judgments about
what is right and wrong, and making decisions about how to pursue con-
crete goals, this person suggested.
T
he discussion turned next to the practical challenges of developing
character in young people. Marvin Berkowitz of the University of
Missouri, St. Louis, drew from research on school-based character
education to identify evidence regarding current strategies and principles
to guide educators and program developers. Reed Larson of the University
of Illinois at Urbana-Champaign, Camille Farrington of the University of
Chicago, and Karen Pittman of the Forum for Youth Investment provided
additional perspectives.
27
people asked about how their traits developed report that they worked to
emulate a parent or other role model, or had determined that they would
not have negative traits they saw in their own parents, such as racism or
dishonesty. He has never encountered a person who reported that his or
her character came from a curriculum or a set of lessons, he commented.
Berkowitz defined character as the complex constellation of psycho-
logical characteristics that motivate and enable individuals to function as
competent moral agents, noting that his definition is essentially the same
as the one Nucci had used. Character education, in turn, he defines as a
way of being through which adult educators and role models foster the
development of character. People are complex organisms, he noted, and
the idea that they can be taught to have character does not fit with the
models of human psychological and moral functioning that developmental
psychologists and other researchers use. The goal of character development
programs should not be to teach, but to promote healthy adult cultures and
actively foster young peoples development, he said.
Berkowitz and his colleagues identified character development strategies
for which there is evidence of effectiveness (Berkowitz at al., 2016). They
reviewed research from the past 16 years that has been collected through
the What Works in Character Education Project,1 meta-analyses and other
recent syntheses of the research, and literature on parenting with respect to
character. The researchers looked for reviews of scientific studies focused on
the outcomes of character education, defined in terms of moral reasoning,
positive psychology, and other frameworksthey did not include effects on
academic achievement or other outcomes. They did not analyze implementa-
tion strategies, but focused on program design and pedagogical approaches
that make a difference.2
Berkowitz developed a structure for thinking about best practices for
character education, which he calls PRIME, for prioritizing character edu-
cation, relationships, intrinsic motivation, modeling, and empowerment.
He used PRIME to organize the primary points he drew from the literature
review conducted for the workshop.
2016]. Berkowitz noted that he and his colleagues are developing the Character Education
Research Clearinghouse (CERCh) to make their findings more widely available. See Berkowitz
et al. (2016) for a detailed discussion of the sources for this work.
2Most of the work they reviewed addressed character education in school settings, but
Berkowitz noted the relevance of this work to out-of-school settings, which have received
significantly less research attention.
Supporting Adolescents
Larson focused on the question of how institutions and caring adults
can best support adolescents grappling with the complex contexts in which
they live. Aristotle, he pointed out, argued that neither rote repetition nor
teaching would prepare a young person to confront a difficult situation,
arguing that, We become just by the practice of just actions, self-controlled
by exercising self-control, and courageous by performing acts of courage.
Confronting a genuinely confusing unstructured situation that poses a
moral challenge, Larson added, can be a profound learning experience.
Larson suggested that after-school programs, especially those for adoles-
cents, provide real-world situations, opportunities, and dilemmas through
which young people can practice in the way Aristotle suggested. He referred
to this type of learning as cycles of practice in context. He noted that his
ideas about after-school character education aligned well with Berkowitzs
findings from in-school character education.
Larson has conducted research with 250 youths from different ethnic
groups, using multiple interviews to analyze their accounts of their social
and emotional learning and character development. This work has identi-
fied some active ingredients that are especially important to programs for
older youth: the opportunity to grapple with challenges, including moral
challenges, investment in meaningful goals, and constructive peer processes.
et al., 2015). As both Nucci and Berkowitz had made clear, she noted,
character development entails a lot of different things happening simulta-
neously. She endorsed the framing Berkowitz et al. offered in their paper
(2016, p. 19), that Ultimately the goal of character education is for chil-
dren and adolescents to become good people, to develop into and act as
agents for good in the world. Hence this is about being people of character
even more than it is about acting good.
Random assignment is not a promising strategy for studying a con-
struct this complicated, or the practices that might promote it, Farrington
noted. It is difficult to isolate specific strategies in order to study their
effects, and much existing research has focused on outcomes other than
character itself. Moreover, she noted, researchers are often compelled
to use short-term behaviors as proxies for the state of being a person of
character, or being on the way to becoming one. Parents and educators
use many strategies and practices in the hope that, over many years, their
efforts will pay off as the child or student becomes an adult of character,
but theres no way to be certain at the time that these methods will bear
fruit, she observed.
Farrington and her colleagues conducted a study of what matters in the
development of successful young people, and how and when they develop
the attributes that are important to success. Their research included a re-
view of relevant literature and extensive interviews with experts from fields
including neuroscience, developmental psychology, social psychology, edu-
cation, workforce development, and sociology. They took a broad view of
what success means: a balance of positive work, family, civic engagement,
and other elements, as shown in Figure 3-2. Figure 3-3 illustrates their view
of the foundation that allows young people to develop successfully. The
outer rings show the three essential elements young people need to develop
by the time they reach adulthood: a sense of agency, an integrated identity,
and a set of competencies. (Farrington noted that these elements continue
to develop throughout adulthood as well.) The inner ring shows specific
components that are the foundation for developing the three e ssential ele-
ments. These elements all contribute to a persons functioning as an indi-
vidual (sense of self) and to his or her functioning in the context of family,
community, and the broader world, she added.
Farrington and her colleagues used a developmental approach to ex-
plore when and how these elements develop. A common research approach,
she noted, is to examine the inputsconditions and practicesthat might
help to develop character. She and her colleagues hoped to learn about the
mechanisms through which these practices result in the development of
character, she explained, so they shifted the focus from what adults do to
what young people do, the kinds of experiences they have and how they
make meaning of those experiences.
Young Adulthood
(Postsecondary, Ages 19-22)
INT
WORK
EG FAMILY
DSETS
R
MIN
Y
AT
NC
E DI
AGE
KN S K I L L S E
OW L E D G
DENT I TY
VA L U E S
HEALTH FRIENDS
&
SE N
LF
EDUCATION - R E G U L ATI O
C CIVIC
OM ES ENGAGEMENT
YO PETENCI
S S
UN CE
G ADU T SUC
L
FIGURE 3-2 Model of young adult success.
SOURCE: Nagaoka et al. (2015, p. 4).
behavior, making connections both with other people and among ideas and
experiences, envisioning the future, and integrating experiences into ones
sense of self.
The role of adults, Farrington concluded, is to provide these oppor-
tunities for action and reflection for young people at every stage of their
development. We are very good at helping kids encounter facts and infor-
mation, and practice skills, she observed, but we tend to skip over most
of the others, though they are very important for learning and character
development.
BOX 3-1
Projects Conducted by the Forum for Youth Investment
Karen Pittman highlighted two projects in which the Forum for Youth Invest-
ment has been involved. Ready by Design is a synthesis of research on young
peoples development. The report identifies critical elementswhich could be
described as components of character, social and emotional skills, noncognitive
competencies, or by other termsthat young people need to thrive.a
The second project, Preparing Youth to Thrive, was a collaboration among
eight exemplary youth organizations working with disadvantaged youth. The part-
ners were asked to think about how to harness what they do, Pittman explained.
They identified, practices they use to help vulnerable adolescents strengthen their
social and emotional skills, and identify ways to formalize these approaches so
they can be used in other settings on a large scale. The results are presented in
a series of reports.b
aSee
http://sparkaction.org/sites/sparkaction.org/files/readybydesign.pdf [December 2016].
bSeehttps://www.selpractices.org/resource/preparing-youth-to-thrive-methodology-and-
findings-from-the-sel-challenge [December 2016].
or teachers who are implementing the practice with young people are ob-
served (in a nonthreatening manner) and given feedback about their work.
When this happens, she explained, the educators and program leaders can
deconstruct the practice and use it to sustain improvement.
Pittman also endorsed the importance of developmental pedagogy, as
Berkowitz had discussed. She suggested that the key question is whether any
given program is aligned with the ideas about how young people develop
over time that were articulated across the presentations. The out-of-school
literature, she observed, echoes much of what Berkowitz had reported from
the in-school literature, and in her view the time is right to merge these.
No matter what system you are working in, she commented, it will be
important to build a culture that supports youth development, and to make
sure all staff understand how to implement the practices aimed at doing this
and integrate them with other goals of the program.
Pittman closed with the analogy of school as a big jar filled with t ennis
balls representing the primary teaching objectives it is responsible for, such
as algebra, language arts, and social studies. The jar is full: there no room
for another tennis ball so adding a requirement for social and emotional
learning, or character education, will seem impossible. Instead of viewing
those goals as another tennis ball, Pittman suggested, educators should view
them as marbles or sand that can be poured into the jar and can fit around
the tennis balls. You may not immediately change what happens within the
tennis balls, she commented, but you begin the process of engaging all
the adults in the school in helping young people develop character.
Investing in Implementation
and Evaluation
T
he discussions of what works in developing character pointed to
questions about how specific practices and strategies can be effec-
tive. The importance of both implementing programs faithfully and
evaluating their effectiveness has been documented in many fields,1 and the
workshop turned next to ideas from both research and practice on these
key elements of character education. Researchers Joseph Durlak of Loyola
University and William Trochim of Cornell University discussed implemen-
tation and evaluation, respectively. Mike Surbaugh of the Boy Scouts of
America and Donald Floyd of the National 4-H Council (retired) provided
input and perspectives from the practitioner perspective.
39
ture on it has developed over the last 20 years, which he reviewed for the
workshop (Durlak, 2016). One point this research has made obvious, in his
view, is that the program you think you are doing almost never turns out
to be the program that actually occurs. His emphasis on the importance
of implementation applies not just to programs, but also to practices or
strategies, as they are carried out in the real world, he added, and is based
on a consistent research finding.
A key reason why programs and practices often look so different from
their designs, in Durlaks view, is the wide chasm between the worlds of
research and practice. Despite good intentions on both sides, he noted, very
few of the many programs and strategies researchers evaluate and find to
be effective are adopted in the world of practice. The results researchers
quantify, he added, rarely translate into comparable degrees of effective-
ness when the ideas are put into practice. There has been little incentive
for researchers to offer practitioners concrete assistance in implementing
their programs, he added, but a focus on implementation can be a bridge
between research and practice. Often, he added, when researchers and
practitioners collaborate in focusing on effective implementation, programs
turn out to be even more effective than researchers had been able to show
on their own.
The research also shows that when attention to the quality of imple-
mentation increases, outcomes improve, Durlak noted. Conversely, poor
implementation can render a program ineffective, he added.
For example, Durlak observed, a meta-analysis of reviews of school-
based programs designed to improve social and emotional learning (target
ing, for example, conduct problems, emotional distress, and antisocial
behavior) showed that results were significantly larger for settings in which
the program was implemented effectively than in settings where it was not
(Durlak et al., 2011). Many of the effect sizes were double or more than
double for the effectively implemented programs, he noted.
ment the program, may have negative attitudes about the program, or may
simply be resistant to change. If these barriers are present and cannot be
overcome, Durlak suggested, the program is not going to work.
However, there are elements that can help organizers facilitate im-
plementation and overcome barriers, Durlak continued. First is effective
leadership and clear allocation of responsibility and lines of accountability.
Another is flexibility within an organization, so leaders can, for example,
reallocate resources or seek external ones in response to unanticipated
needs. Coordination across all the entities involved, including local or state
agencies or other governmental bodies, is also valuable, Durlak pointed
out. Character education programs are usually components of policies
intended to affect more than a single program, so the way the policy is
implemented is just as important as the implementation of a particular
program or practice.
The staff who are implementing a program or practice will need ongo-
ing feedback, Durlak noted, so they can adapt and improve as they proceed.
The feedback should be offered in a collaborative way, together with sup-
port for assessing the feedback and figuring out how to respond. Programs
can use systems for tracking data and efficiently sharing feedback to make
this part of the process more systematic, he added. Ongoing research can
then build on the feedback by systematically examining what can be learned
from the implementation experience in a particular context.
We dont have a nationwide structure for implementation, Durlak
pointed out, even though it is important in virtually every sector in which
evidence-based interventions are used. The bridge between research and
practice Durlak called for requires an implementation system that includes:
and staff at local organizations, trainers and consultants, and the young
people who are to receive the services and their families all have a profound
interest in the outcome and also have ideas to contribute. Sadly, he noted,
these five groups rarely collaborate, and it is primarily individual founda-
tions that make such collaboration possible in the United States when it
does occur. Other countries, he pointed out, have institutionalized systems
for implementation and evaluation. Effective implementation cannot be
rushed, he observed. Some programs are quite complicated and require
significant change in an organization and time to develop. It is only worth
pursuing programs, in Durlaks view, if the resources to support effective
implementation are available.
ADVANCING EVALUATION
Many character education programs have never been formally evalu-
ated, William Trochim pointed out, and we still dont know if and how
they work. Despite the existence of resources such as the What Works
Clearinghouse (see Chapter 3), there is very little formal evidence for pro-
gram designers and practitioners to use, he noted. The reason in his view
is that we dont have the time and resources to invest in evaluation. It is
often one of the lowest priorities within organizations, and few have staff
with the expertise to conduct sound evaluations, he said.
He collaborated with workshop steering committee member Jennifer
Brown Urban to apply findings from recent scholarship linking evaluation
in other fields to the character education context (Trochim and Urban,
2016). Their objective was to offer a new way of thinking about evaluation,
which Trochim described as an evolutionary systems approach.
An Evolutionary Approach
Trochim began with the connection to the theory of evolution, based
on the idea that programs are like organisms. Like a potato, he suggested,
a program is one in a population of varieties within the same species. The
theories that guide the programstated or assumed expectations about
what the program will do and howare the essential instructions for
the program, akin to the genetic code that instructs an organism how
to d evelop. The inevitable variations in the way the program is imple-
mented each have consequences. In subsequent generations of the program,
Trochim said, natural selection determines which of these variations con-
tinue and which are discarded.
Trochim did not intend this model only as a metaphor, he noted: he
argued that programs evolve by the same rules that govern the biological
process of natural selection. To elaborate, he noted that the potato was not
native to Europe but when a single variety was introduced it soon became
a dietary staple, particularly in Ireland. A blight infected this variety during
the 1840s, and because Ireland relied so heavily on it there was widespread
famine. Trochim suggested that there is a parallel danger for character edu-
cation in saying we know what worksjust do that.
Natural selection in the context of programs or interventions, he con-
tinued, is a process of blind variation and selective retention. Blind varia-
tion, he explained, does not mean that those who implement programs are
acting without reason, but rather that they make choices without having
information to indicate whether they will work or not. At the same time,
he added, people in these situations do draw on experience and knowledge
in making their decisions, even if it does not come from formal evaluations.
Thus they selectively retain particular approaches based on many factors,
including personal, social, and sociopolitical influences.
Charles Darwins purpose in exploring the way organisms develop,
Trochim continued, actually grew out of an interest in artificial selection,
the breeding of plants and animals to produce new ones that have desired
traits. Trying to develop character using various interventions, he suggested,
is another form of artificial selection. We are breeding ideas and trying to
see which ones will survive, he said.
Evaluation plays a critical role in this endeavor, in Trochims view,
because it is the basis for judging whether or not an intervention worked
and should be selected. It also plays a role in the development of variations
to be tested, he explained: We want a system that generates varieties [of
ideas] and also allows us to judge and select them. Every program has a
life cycleit progresses through phases or stagesand over time programs
and theories evolve. Eventually, if one looks at character development pro-
grams across time, it is possible to trace the family tree of the programs
and the theories that guide them.
Programs evolve in the way organisms do in part because their develop-
ment can be compared to human development, Trochim pointed out. Thus,
another key to the approach to evaluation that he and Urban advocate is
relational developmental systems theory, which, as Schmid Callina had
explained, focuses on the reciprocal influences that individuals and their
contexts have on one another (see Chapter 2). Just as individuals have
the potential to adapt across the life span, Trochim explained, programs
develop and are adapted in the context in which they are implemented.
Trochim observed that Darwin never used the phrase survival of the fit-
test; his argument was that it is the species that are best able to respond
to changes in their environments that survive, not the strongest or most
intelligent.
Systems Thinking
Thinking about evaluation has increasingly come to include a focus on
the idea of systems, the interaction of many parts, Trochim explained. The
interactions between the individual organism (or program) and its envi
ronment occur in the context of nested systems, and effective evaluation
requires understanding of how the relevant systems interact. This is a rela-
tively new application of an ancient idea, Trochim noted. Many philoso-
phers and scholars have developed systems-based theories, or ideas about
the way systems observed in nature or society are organized and structured.
Trochim highlighted elements that are particularly relevant to evalua-
tion. One is that a program is a whole that is greater than the sum of its
multiple parts. These parts contribute to both static processes, those that
tend to operate in linear, predictable ways, and dynamic ones, which are
not predictable. Like an individual, he added, a program functions within
a hierarchical nest of contexts, from the personal, local, and regional,
through the national and global. Each stakeholder reflects these contexts
in a different way and will have a different perspective on the program, its
goals, and its boundaries. For example, practitioners may focus on engaging
individual young people, while a funder may focus on broad societal goals.
DISCUSSION
In discussion participants focused on questions about how youth orga-
nizations can respond in a practical way to new ideas. Several participants
commented on the difficulties of collaborating across organizations. There
has certainly been change, one noted, in that 25 years ago it would have
been astonishing to see a program such as Imagine Science that involved
true collaboration across major organizations. Allowing independent
researchers the access they need to conduct studies of their work was also
rare for such organizations, but it is difficult to see how widespread these
sorts of changes are, this person added.
Another noted that funders may hold the key. They encourage their
grantees to collaborate and to evaluate their work, this person noted, but
they seldom offer funding for these purposes. They are in a unique position
to promote collaboration and the sharing of both tested experience and in-
novative ideas. For example, a funder could work with the grantees under
a particular funding portfolio to identify commonalities and facilitate the
sharing of ideas. The large national organizations can also play a leader-
ship role, as the discussions of 4-H and the Boy Scouts illustrated, noted
another. The true competitors for youth organizations are not other youth
organizations, another person commented, but video games, smart phones,
and the many other kinds of activities that occupy young peoples time. Its
the kids who are not involved in any organized youth activity we need to
reach, he added.
S
taff and leaders are at the heart of every youth program. Recruit-
ing and retaining experienced, qualified individuals and giving them
the support and training they need is a primary challenge for every
organization. Deborah Moroney of the American Institutes for Research
explored research on the characteristics of the workforce that serves in
out-of-school youth programs and offered ideas for building their capacity
to support positive development. Noelle Hurd of the University of Virginia,
Rob Jagers of the University of Michigan, and Mary Keller of the Military
Child Education Coalition offered their perspectives on the role of culture
and context in the development of supportive relationships between young
people and program staff.
51
Challenges
This is a lot to do for a workforce that is not compensated well and
does not have a traditional educational pathway, and in which there is
high turnover, Moroney pointed out. Yet youth workers want very much
to contribute to young peoples social and emotional development, she
said. Surveys have shown that youth workers are hesitant about how well
prepared they are to do this work and are eager for more training and
guidance in this area.
Moroney acknowledged that the complexity of the overlapping frame-
works for thinking about this kind of development can be daunting for
program staff and leaders. She said she hopes program leaders and staff
will recognize that whatever conceptual focus is adoptedsuch as char-
acter education or social and emotional learningthe mission of positive
development is one that has always been part of the reason out-of-school
youth programs exist. Many youth workers eyes glaze over when they
hear about another new thing, Moroney commented, and we need to
make sure this is not another new thing. New ideas about how to sup-
port young peoples positive development should contribute to the way
staff think about all the activities and strategies they already pursue, not
replace them.
Moroney sought the views of research and practice leaders in the field
on how youth workers can best be supported in pursuing the mission of
promoting positive development. They agreed on the importance of several
points, particularly organizational support for professional development.
She noted the importance of learning from other fields about this area. For
example, research on workers and the workplace across many fields and
careers has increasingly focused on the importance of social and emotional
and other noncognitive skills. This research has suggested that supporting
all adults in their career and developmental pathways benefits both workers
and employers, Moroney noted.
The leaders she consulted did not agree on how to address the com-
plexity of and overlap among concepts of character or positive develop-
ment. Practice leaders argued for identifying a simple framework that
could be widely applied, incorporating common elements from existing
ones. Researchers, Moroney noted, did not. She suggested that from a
common-sense perspective, a common framework and vocabulary would be
especially helpful in an environment where there is significant job mobility.
It is not uncommon for youth workers to work at more than one organiza-
tion at the same time, for example, and it would be much easier for those
workers, as well as those who change jobs frequently, to become familiar
with a single framework and develop expertise with it over time, than to
shift gears with every move.
Moroney closed with ideas about next steps for research. The indi-
viduals she consulted particularly emphasized the value of implementation
studies, data on the characteristics of the youth workforce, and the impacts
the practices designed to foster positive development have in young peoples
later lives.
plained. Training and ongoing support are key to helping youth workers
examine their own potential biases and adopt cultural humilityan open-
ness to and curiosity about other cultural backgrounds and experiences
Hurd commented, particularly when the adults and young people are
bridging differences in race, ethnicity, gender, or socioeconomic experience.
Youth development programs aim to build on young peoples strengths,
Hurd noted, and this is also particularly important for programs that serve
marginalized youths. Young people in challenged communities can derive
long-term benefits when a trusted adult helps them to identify strengths in
their own character, as well as assets in their homes and schools.
1He defined oppression as asymmetrical power relations that are reflected in deprivation,
discrimination, exclusion, and exploitation, and explained that it can be internalized by op-
pressed groups, who act in ways that run counter to their own best interests as a result.
2Jagers noted that social justice has many meanings and that he was focused primarily in this
More specifically, he added, the ways adults interact with young people
could be understood as a continuum. At one end is a relationship in which
the adult is in complete control of the situation and treats the young person
as a vessel into which knowledge and wisdom are poured. In the middle
would be a situation in which the adult and the youth share control, and
young people have an active role and voice in what happens. At the other
end would be a relationship in which youth are autonomous and have total
control of the situation.
In his own work with young people, Jagers noted, he is guided by
the idea that young people need to experience autonomy. Out-of-school
programs are ideal settings for them to identify and analyze challenges for
themselves, and to plan and implement their responses, with the thought-
ful guidance of trained adults. He encouraged participants to explore the
Wolverine Pathways program at the University of Michigan, which exempli-
fies this approach.3 This program is designed to boost the academic, social,
and cultural preparation of low-income Michigan high school students to
increase their chances to enroll at the University of Michigan and complete
a degree there. The program focuses on helping young people take on the
role of change agents, to improve the well-being of their communities.
Military Children
Mary Keller used the example of military children to focus on the chal-
lenges that face highly mobile children and the ways youth programs can
support them. There are currently approximately 2 million children whose
parents are serving in some way in the armed forces, she noted. These chil-
dren move an average of once every 2 to 3 years because of their military
parents responsibilities, Keller noted, and this means they change schools
often. Only approximately 8 percent are enrolled in Department of Defense
schools, she added, yet all of these children need consistent and reliable
programming.
Frequent moves affect childrens academic trajectories, Keller com-
mented, and also their social and emotional learning. Many of these chil-
dren are also exposed to significant stress and trauma. For example, there
is a population of nearly 2 million children whose parents are veterans of
conflict that took place after September 11, 2001. Children and families
often continue to have stresses related to a parents military service even
after the parent leaves the military, she added, noting that female veterans
with children are a growing segment of the homeless population in the
United States. Positive youth development is very important for these over-
lapping populations of children, Keller emphasized.
DISCUSSION
Many comments addressed the differences between and overlap among
the ways researchers have framed the objectives of building character or
helping young people develop in a positive way. A focus on the differences
is a giant risk to the entire endeavor we all care so much about, one
participant said. In his view it is critical that practitioners and researchers
build on what we know rather than swapping out one trend for another.
Several people expressed agreement. One suggested that supporting young
peoples growth is a consistent theme across all of the frameworks. Another
commented that the DNA [of the frameworks] is the sameno one wants
The last point on that issue was that research-practice partnerships are of
real value in addressing this challenge. They help researchers understand
how their ideas function in practice and they also empower practitioners
to think more broadly about what they are doing.
Other comments focused on the risk that youth programs designed
to build character strengths may reinforce social conformity. We pat
ourselves on the back for our commitment to democracy and equal op-
portunity even though both are very imperfect in the United States, one
person commented. He welcomed the reminder to question assumptions
and implicit biases. No youth program should encourage young people
to take their assumed place, e.g., a low-skill job, this person noted. Youth
programs should be places where adults help young people demand a re-
shaping of society, in this persons view.
Another participant noted that programs that work with underserved
young people too often have a deficit/problem model and focus on what
it will take to help them succeed. Following up, another person pointed to
the importance of recognizing the role of power and privilege in relation-
ships between youth and adults. In that vein, another noted that though
these subjects can sometimes be uncomfortable, youth program activities
naturally present countless teachable moments in which well-prepared
adults can bring these broader issues in from the margins.
Thoughtful training and support for youth workers are critical to help
them create a safe space for courageous conversation, another person
noted. This kind of training is needed at every level, this person added, and
should be treated as a basic, on par with the criminal background check.
Making sure that program leaders and staff have this support should be a
priority for those at the policy and funder levels who hope to have broader
Measuring Character
R
esearchers, programs, organizations, and funders all want trust
worthy evidence that young people who are exposed to some form
of character education grow or develop as a result of that experi-
ence. As the rich debate about how to define the nature of character and
goals for character education suggests, however, measuring outcomes in this
area is challenging. Noel Card of the University of Connecticut reviewed
methodological challenges in the measurement of character and proposed
suggestions for addressing them. Discussants Clark McKown of Rush Uni-
versity Medical Center and Nancy Deutsch of the University of Virginia
offered two additional perspectives.
METHODOLOGICAL ISSUES
The divide between research and practice is only part of the challenge
for improving practice in the development of character and social and emo-
tional learning, explained Card. There is also a gulf between methodology
and research and practice, he suggested, and he began with a practical,
hands-on overview of how key psychometric principles apply to the study
of character development.
Psychometric Fundamentals
Reliability, validity, and equivalence are three fundamental principles
of measurement, Card noted, and each presents challenges for the mea-
63
Reliability
Reliability is a measure of how repeatable a measure is, or the ex-
tent to which scores are consistent when a test of a particular construct is
given multiple times, Card explained. There are several ways to think about
reliability, and most common is the idea of internal consistency reliability.
Typically, test developers want to measure a construct in more than one
way, using multiple items or tasks. For example, he commented, a survey
or interview might include multiple questions about an aspect of character
to get at it in different ways. Measures of internal consistency indicate
how repeatable these different questions arethe degree to which they
are measuring the same thing, based on how similar the scores are across
the group of questions. Psychometricians tend to turn up their noses at
this type of reliability, Card explained, because it is based on assumptions
about how parallel the questions truly are.
Another measure, testretest reliability, is an indication of how repeat-
able the test is across multiple measurement occasions. In this case, the
assumption is that the construct itself is stable over a particular time spanso
if the time span between testing and retesting is too long it would be possible
that any change is accounted for not by the way the measure functions but
by actual developments in the person being tested.
A third type of reliability is inter-informant reliability, a measure of how
repeatable test results are across multiple individuals who report on the
character construct being assessed. In this case, Card explained, the under-
lying assumption is that the character construct being measured is the same
in different contexts in which the reporters observe it. For example, both a
classroom teacher and an after-school counselor might be asked to evaluate
a childs character based on observation. If the character construct is stable
across the classroom and after-school setting, then any differences in the
observations would contribute to a lower inter-informant reliability score.
It is worth noting, Card observed, that we dont really know how
stable elements of character may be over time or across different settings.
Nevertheless, internal consistency is important. It is also easier to collect
evidence about multiple questions within a single test than to collect evi-
dence over time or across settings, so this measure is commonly used in
the context of character education. It is important not to overemphasize
the value of reliability measures, Card pointed out. All of the measures are
estimates from a sample, so they are measures of how a particular testing
instrument performs in a particular context at a particular time.
MEASURING CHARACTER 65
Validity
Validity and equivalence are at least as important as reliability, Card
continued. Validity is the extent to which a measurement actually assesses
what it is intended to assess. Suppose one wanted to measure how fre-
quently young people were displaying a particular prosocial behavior, and
differences among them, Card suggested. The challenge is to find a way
to measure the behavior accurately without inadvertently measuring other
things that are not relevant. One could use a scale that measures a very spe-
cific behavior, such as a questionnaire item asking how often young people
offer to help the teacher, but that might not provide enough information,
he pointed out. On the other hand, if the construct is not carefully defined
the results might reflect the respondents understanding that the behavior
is socially desirable, rather than how well they display it.
The key to validity, Card emphasized, is to have a very clear definition
of the character or development being studied, but the field has not settled
that problem. We have to work toward this, he suggested, but it is still
possible to measure character without either assuming that it is immutable
or reducing the complexity of human behavior, psychology, and cognition
to a simple variable. What measurement experts can do instead is to define
a domain they can measure that represents the traits, skills, or behaviors
being studied, as shown in Figure 6-1.
The circumference of the circle represents the boundaries of an opera-
tional definition of the construct to be measured, such as a set of behav-
iors that are characteristic of the prosocial attitude being tested: the dots
represent items or questions that sample elements of this domain, perhaps
particular behaviors. The dots that lie outside the circle assess behaviors or
attitudes that are outside of the defined domain, such as a belief that the
teacher favors particular behaviors.
At the center of the circle is the bullseye, Card explainedthe heart
of the construct. Theoretically there could be an infinite number of dots
inside the circle, questions that accurately measure some aspect of the
domain. The job of the test developer is to identify a practical number of
them that, together, provide an accurate composite picture of the domain
being tested. The arrows across the bottom circle represent the way a
carefully chosen set of items will triangulate around the bullseye, Card
explained.
The distribution of the dots in Figure 6-1 represents a situation in
which there is relatively low internal consistency reliability. If the researcher
removes the items that contribute most to this low internal consistency,
Card explained, the result might be a set of items that yield highly consis-
tent results but lie at the edge of the circlenot centered around the bulls-
eye. In other words, efforts to improve reliability might produce a highly
reliable measure of the wrong thing, Card emphasized, and this danger is
another reason it is so important to have a clear definition of the construct.
Equivalence
The third key psychometric idea is equivalence, Card went on, an in-
dicator that the measurement performs in the same way across groups or
time. This idea is useful in assessing test fairness, or differential item func-
tioning, because it is the basis for determining whether a test item performs
differently across groups, such as males and females or ethnic groups. That
is, an item performs differentially if it yields different results for individuals
who in reality have comparable knowledge or skills only because something
about the way it is designed favors one individual over the other. Items that
are equivalent across time allow researchers to accurately assess changes
that result from an intervention, for example. There are three levels of
equivalence, Card noted (configural, weak, and strong), but he suggested
MEASURING CHARACTER 67
Denition #2
Denition #1
the diversity of the adults and young people involved, is striking. Card
suggested that those involved in measurement of character and related
constructs should pay greater attention to the possibility that psychometric
properties may function differently across diverse populations and contexts.
It is important to assess these psychometric properties in every study, he
emphasized, and to pay explicit attention to measuring equivalence.
Moreover, a researcher studying the impact of an intervention needs
to consider the possibility that other developments that occur at the same
time the intervention is implemented may affect the measure. Box 6-1 lists
the kinds of errors that can arise when this is not done effectively. The
bottom line, Card suggested, is that we cannot have confidence that an
intervention was effective without establishing that it affected the aspect of
character itself, rather than the measurement of it.
Two Examples
Card offered two examples to illustrate these ideas, studies of gratitude
and humility that were parts of an ongoing meta-analysis of 11 character
strengths he is conducting with colleagues. They reflect two very differ-
ent situations, he noted. He and his colleagues have found that studies
of gratitude all tend to use one of a very small number of measurement
instruments. One of these is a six-item questionnaire that was developed
through a series of steps. The researchers who developed this test, known
as the GQ-6 (McCollough et al., 2002), first administered a much longer set
of questions to college undergraduates and then used statistical techniques
BOX 6-1
Possible Results of Measurement
Errors in Intervention Studies
MEASURING CHARACTER 69
to identify the six questions with the highest inter-informant reliability and
construct validity.1 They then administered the six-item test to a broader
sample of adults and found similar evidence of construct validity.
A second method for measuring gratitude is the Gratitude, Resent-
ment, and Appreciation Test (GRAT) (Watkins et al., 2003). In this case
the researchers conducted the psychometric analysis using a 55-item test,
which they also administered to college undergraduates. Using statistical
analysis they identified the three factors that appeared to be most relevant
and used those to winnow the list of test questions. They did not, however,
replicate the analysis that identified the three factors, Card noted, though
they did test the psychometric properties.
There were limitations to both of these studies, Card pointed out: a
possible overemphasis on reliability in the first case, and possible errors
in the identification of the relevant factors in the second. His point was
not to criticize the researchers in either caseindeed, he noted that both
were impressive efforts to translate a theoretical concept into a usable
measure. However, both are treated as seminal studies, he commented, and
researchers who later adopted these measures are unlikely to have assessed
the psychometric decisions for themselves or taken them into account in
using the results they obtained using these instruments.
This is not at all uncommon, Card noted: developing sound measures
takes time because it is challenging. Even where there are a few seminal
papers, he pointed out, we dont just put up the victory flag and say done.
Most instruments need to be modified as theory changes, and to be adapted
for use with particular populations and in particular contexts, he added. In
the case of gratitude, Card noted, there are actually four popular measures
that have been used in a diverse set of studiesfindings that emerge across
a body of work that uses different methods can yield valuable conclusions.
Researchers of humility, however, do not have a small number of widely
used measures to rely on, and many seem to develop their own unique mea-
sure, to be used in that study and never again. One problem with this
situation is that any two studies of humility may actually be measuring very
different things. Conversely, other studies may be measuring constructs that
are not identified as humility, but which match definitions of it that other
people use. Some researchers may develop a measure without adequately
testing its psychometric properties, Card added, which may be one reason
why so many researchers in this area have chosen not to rely on the previ-
ous literature.
1There are several categories of validity; construct validity is the type Card had described,
Suggestions
Card closed with a few suggestions with respect to planning studies,
reporting results, and accumulating a body of trustworthy research. He
challenged participants to include questions about psychometrics in their
assessments of available research. It is important to explore what sorts of
measures were used to produce compelling results, how they were devel-
oped, and what limitations might need to be taken into account. He em-
phasized that measuring changes in character attributes is not the same as
simply measuring that attribute at a particular time.
Card also noted that We shouldnt assume that the creator of an
instrument has thought about every population and context in which it
might be used. Others who use a measure need not be rigid, he suggested,
but should carefully modify it to better suit different circumstances. Pairing
a quantitative measure with qualitative or mixed-method studiespossibly
working with a multidisciplinary teamis another important strategy for
improving the definition of the construct and the associated measure.
Finally, Card observed that full reporting on psychometric properties
should be a part of any study, so that readers can fully understand what
was done and what possible limitations they should consider. He acknowl-
edged that this is difficult. Some of the analyses are technically demanding,
which means time, expertise, and resources are needed to support that extra
effort. Studies that include detailed psychometric analysis tend to be less
interesting to publishers, he noted, but it is important that this information
be available, perhaps as a component of a larger study.
We need to shift perceptions, Card suggested, so that people in the
field value psychometric analysis and dont just view it as a preliminary
to more interesting results. Meta-analysis of a body of research results is
an important tool for identifying findings that reflect a much larger sample
than any single study could, as well as a more diverse set of study settings
and types of measures. Because studies typically differ in many ways,
Card noted, it is possible to use statistical procedures to identify causes of
some of the differences that emerge from a synthesis of results across a large
body of studies. This sort of information can help the planners of future
studies to make well-supported decisions about the measures they will use.
Card closed with the point that if there were easy solutions we would
have figured this out long ago. He pointed out that his focus was on
MEASURING CHARACTER 71
MEASURING CHARACTER 73
MEASURING CHARACTER 75
Workshop Themes
T
he final session was an opportunity for participants and presenters
to sift through the ideas that emerged in the 2 days of discussion.
Participants reported on the discussions they had had in breakout
sessions and their responses to the workshop framing questions (see Chap-
ter 1 and Appendix D). The workshop closed with concluding thoughts
from committee members.
Defining Character
Character is not a trait and does not break down into easily mea-
surable components. Nevertheless, it is reassuring to learn that
there is constancy in the theoretical background for current ideas
of character and social and emotional learningthat we really are
a field.
Policy makers, researchers, and practitioners all need to act in a
more coordinated way, but the lack of a shared language is an
obstacle. Coordination could help all concerned avoid misunder-
standings and make the available information more effective.
77
WORKSHOP THEMES 79
CLOSING THOUGHTS
The workshop ended with closing thoughts from several committee
members.1
Lucy Friedman noted that the workshop discussion did not focus much
on what is known not to work, but that there is evidence on this as
well. Scaring young people and pouring information into them are both
ineffective, for example, she said. This is an exciting time in the field, in
her view. She sees researchers and practitioners coming together to change
the conversation and help gain recognition for the importance of out-of-
school time. Continued developments in research are necessary to this, she
observed.
Jennifer Brown Urban noted that the available workshop time did not
allow William Trochim to describe the practical, how-to elements of
their work, but she encouraged participants to use the paper and project
website as resources.2 She also commented that collaboration is challenging
for organizations, and that the diversity in out-of-school opportunities is
a good thing. We need as a community to provide as many opportunities
as possible so kids can find what meets their needs and interests, she said.
This diversity makes research challenging, but the individuality of programs
is often what makes them work, she observed.
Catherine Bradshaw reinforced the message that measurement tools
beyond surveys are important, and that researchers need to consider what
can be learned from programs with a wide variety of missions and defini-
tions of familiar objectives. If a program does have an impact on character,
she wondered, can it count as character development even if that is not a
part of its philosophy?
Ellen Gannett commented that she as a practitioner feels proud of the
efforts her colleagues have made to join forces with researchers and also
to articulate their own ideas and questions. Building a skilled and stable
workforce has become a high priority in the field, and she said she sees the
1Not all of the committee members were able to be present for this final discussion.
2See https://core.human.cornell.edu/research/systems/protocol [December 2016].
effects of that and of the many contributions from research. The out-of-
school workforce, she suggested, stands shoulder to shoulder with other
professionals, and is on its way to further growth
Richard Lerner thanked the participants for their contributions to
a rich discussion of the many dimensions of the challenge. The work-
shop enhanced his understanding of the complex issues and particularly
reinforced the importance of making cultural diversity and social justice a
predominant focus in both research and practice, he noted. The primary
goal for character education under any name is to make the world b etter
for young people, he said, and the passion for doing better work on
behalf of young people through research and practice is a noble purpose.
References
Berkowitz, M., Bier, M., and McCauley, B. (2016). Effective Features and Practices that Sup-
port Character Development. Paper prepared for the Workshop on Approaches to the
Development of Character, July 26, The National Academies of Sciences, Engineering,
and Medicine, Washington, DC. Available: http://sites.nationalacademies.org/cs/groups/
dbassesite/documents/webpage/dbasse_173494.pdf [December 2016].
Berry, J.W. (1997). Immigration, acculturation, and adaptation. Applied Psychology, 46(1),
5-34.
Bronfenbrenner, U. (1979). The Ecology of Human Development: Experiments by Nature and
Design. Cambridge, MA: Harvard University Press.
Buckley, J., Archibald, T., Hargraves, M., and Trochim, W.M. (2015). Defining and teaching
evaluative thinking: Insights from research on critical thinking. American Journal of
Evaluation, 36(3), 375-388.
Card, N.A. (2016). Methodological Issues in Measuring the Development of Character. P aper
prepared for the Workshop on Approaches to the Development of Character, July 26,
The National Academies of Sciences, Engineering, and Medicine, Washington, DC.
Available: http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/
dbasse_173818.pdf [December 2016].
Damon, W. (2009). The Path to Purpose: How Young People Find Their Calling in Life. New
York: Simon and Schuster.
Duckworth, A. (2016). Grit: The Power of Passion and Perseverance. New York: Simon and
Schuster.
Durlak, J. (2016). What You HAVE to Know aboutProgram Implementation. Paper prepared
for the Workshop on Approaches to the Development of Character, July 26, The National
Academies of Sciences, Engineering, and Medicine, Washington, DC. Available: http://
sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_173421.pdf
[December 2016].
Durlak, J.A., Weissberg, R.P., Dymnicki, A.B., Taylor, R.D., and Schellinger, K.B. (2011). The
impact of enhancing students social and emotional learning: A meta-analysis of school-
based universal interventions. Child Development, 82(1), 405-432.
81
Emmons, R.A. (2009). Greatest of the virtues? Gratitude and the grateful personality. In D.
Narvaez and D. Lapsley (Eds.), Personality, Identity, and Character: Explorations in
Moral Psychology (pp. 256-270). New York: Cambridge University Press.
Fehr, B., Sprecher, S., and Underwood, L.G. (Eds.). (2009). The Science of Compassionate
Love: Theory, Research, and Applications. Hoboken, NJ: John Wiley & Sons.
Gordon, M. (2005). Roots of Empathy: Changing the World Child by Child. Toronto: Thomas
Allen Publishers.
Hartshorne, H., and May, M.A. (1928). Studies in Deceit. New York: Macmillan.
Keltner, D., and Haidt, J. (2003). Approaching awe, a moral, spiritual, and aesthetic emotion.
Cognition & Emotion, 17(2), 297-314.
Kohlberg, L. (1984). Essays on Moral Development. Vol 2: The Psychology of Moral Develop-
ment. San Francisco: Harper and Row.
Kohlberg, L., and Mayer, R. (1972). Development as the aim of education. Harvard Educa-
tional Review, 42(4), 449-496.
Lapsley, D. (1996). Moral Psychology. Boulder, CO: Westview.
Larson, R. (2016). Learning Character. PowerPoint prepared for the Workshop on Approaches
to the Development of Character, July 26, The National Academies of Sciences, Engi-
neering, and Medicine, Washington, DC. Available: http://sites.nationalacademies.org/cs/
groups/dbassesite/documents/webpage/dbasse_173631.pdf [December 2016].
Lerner, R.M., Dowling, E.M., and Anderson, P.M. (2003). Positive youth development:
Thriving as the basis of personhood and civil society. Applied Developmental Science,
7(3), 172-180.
McCullough, M.E., Emmons, R.A., and Tsang, J.-A. (2002). The grateful disposition: A
conceptual and empirical topography. Journal of Personality and Social Psychology, 82,
112-127.
McGrath, R.E. (2016). Essential Virtues. PowerPoint prepared for the Workshop on Ap-
proaches to the Development of Character, July 26, The National Academies of Sciences,
Engineering, and Medicine, Washington, DC. Available: http://sites.nationalacademies.
org/cs/groups/dbassesite/documents/webpage/dbasse_173629.pdf [December 2016].
McKown, C., Russo-Ponsaran, N.M., Allen, A.A., Johnson, J., and Russo, J. (2016). Web-
based direct assessment of childrens social-emotional comprehension. Journal of Psycho-
educational Assessment, 34, 322-338. DOI: 10.1177/0734282915604564.
Moroney, D. (2016). The Readiness of the Out-of-School Time Workforce to Intentionally
Support Participants Social and Emotional Development: A Review of the Literature and
Future Directions. Paper prepared for the Workshop on Approaches to the Development
of Character, July 26, The National Academies of Sciences, Engineering, and Medi-
cine, Washington, DC. Available: http://sites.nationalacademies.org/cs/groups/dbassesite/
documents/webpage/dbasse_173641.pdf [December 2016].
Nagaoka, J., Farrington, C.A., Ehrlich, S.B., and Heath, R.D. (2015). Foundations for
Young Adult Success: A Developmental Framework. Chicago, IL: The University
of Chicago Consortium on Chicago School Research. Available: https://consortium.
uchicago.edu/publications/foundations-young-adult-success-developmental-framework
[December 2016].
Nucci, L. (2016). Character: A Multi-faceted Developmental System. Paper prepared for
the Workshop on Approaches to the Development of Character, July 26, The National
Academies of Sciences, Engineering, and Medicine, Washington, DC. Available: http://
sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_173496.pdf
[December 2016].
Peterson, C., and Seligman, M.E.P. (2004). Character Strengths and Virtues: A Handbook
and Classification. New York: Oxford University Press and Washington, DC: American
Psychological Association.
REFERENCES 83
Roeser, R., Vago, D., Pinela, C., Morris, L., Taylor, C., and Harrison, J. (2014). Contemplative
education: Cultivating ethical development through mindfulness training. In L. Nucci, D.
Narvaez, and T. Krettenauer (Eds.), Handbook of Moral and Character Education (2nd
ed.) (pp. 234-247). New York: Routledge.
Rose, T. (2015). The End of Average: How We Succeed in a World That Values Sameness.
New York: HarperCollins.
Rumbaut, R.G., and Komaie, G. (2010). Immigration and adult transitions. Future Child,
20(1), 43-66.
Santrock, J.W. (2007). Child Development. Eleventh edition. New York: McGraw-Hill Com-
panies, Inc.
Seligman, M.E. (2004). Authentic Happiness: Using the New Positive Psychology to Realize
Your Potential for Lasting Fulfillment. New York: Simon and Schuster.
Snyder, C.R. (2002). Hope theory: Rainbows in the mind. Psychological Inquiry, 13(4),
249-275.
Sokol, B.W., Hammond, S.I., and Berkowitz, M.W. (2010). The developmental contours of char-
acter. In T. Lovat, R. Toomey, and N. Clement (Eds.), International Research Handbook on
Values Education and Student Wellbeing (pp. 579-603). Dordrecht, N etherlands: Springer.
Stepick, A., Stepick, C.D., and Labissiere, Y. (2008). South Floridas immigrant youth and
civic engagement: Major engagementminor differences. Applied Development Science,
12(2), 57-65.
Surez-Orozco, C. (2016). The Role of Culture and Context on Character Development.
PowerPoint prepared for the Workshop on Approaches to the Development of Character,
July 26, The National Academies of Sciences, Engineering, and Medicine, Washing-
ton, DC. Available: http://sites.nationalacademies.org/cs/groups/dbassesite/documents/
webpage/dbasse_173633.pdf [December 2016].
Surez-Orozco, C., Hernandez, M.G., and Casanova, S. (2015). Its sort of my calling: The
civic participation and social responsibility immigrant origin emerging adults. Research
in Human Development, 12(1), 84-99.
Trochim, W.M., and Urban, J.B. (2016). Advancing Evaluation of Character Building
Programs. Paper prepared for the Workshop on Approaches to the Development of
Character, July 26, The National Academies of Sciences, Engineering, and Medicine,
Washington, DC. Available: http://sites.nationalacademies.org/cs/groups/dbassesite/
documents/webpage/dbasse_173497.pdf [December 2016].
Tudge, J., Frietas, L.B., and O-Brien, L.T. (2015). The virtue of gratitude: A developmental
and cultural approach. Human Development, 58, 281-300.
Watkins, P.C., Woodward, K., Stone, T., and Kolts, R.L. (2003). Gratitude and happiness:
Development of a measure of gratitude, and relationships with subjective well-being.
Social Behavior and Personality, 31, 431-452.
Appendix A
Workshop Agenda
TUESDAY, JULY 26
8:309:00 Introductory Remarks and Setting the Stage
85
10:3010:45 Break
12:151:15 Lunch
APPENDIX A 87
WEDNESDAY, JULY 27
8:459:00 Goals for the Day
Deborah Vandell
11:0011:15 Break
12:151:00 Lunch
APPENDIX A 89
2:302:45 Break
What is character?
What works in developing character?
Investing in implementation and evaluation
Open Discussion
4:30 Adjourn
Appendix B
Participant List
91
APPENDIX B 93
Appendix C
Biographical Sketches of
Committee Members and Presenters
95
APPENDIX C 97
interest, her book Failing at School: Lessons for Redesigning Urban High
Schools documents how high schools systematically construct widespread
student failure for the most socially vulnerable students, and offers practi-
cal recommendations. She draws on 15 years experience as a public high
school teacher and National Board Certified Teacher Mentor. She received
a B.A. from the University of California, Santa Cruz, teacher certification
from Mills College, and a Ph.D. in policy studies in urban education from
the University of Illinois at Chicago.
Donald Floyd (Presenter) recently retired from his role as CEO and presi-
dent of the National 4-H Council. He has been involved with the leader-
ship of youth-serving, nonprofit organizations for more than 35 years. For
17 years, he held local- and national-level positions, including national
executive vice president of Junior Achievement. He recently completed
a term as chair of the National Collaboration for Youth and is currently
a trustee with the Americas Promise Alliance. He served as a trustee of
Albright College and was secretary of the board, member of the executive
committee, and vice chair of its governance committee. He is former chair
of the International Leaders Committee of the Institute for Applied Re-
search in Youth Development, Tufts University. He was one of six inaugural
recipients of the International Fellows in Applied Developmental Science.
He has traveled to 35 countries to establish youth programs and has lived
in Japan and Ghana. He served in the U.S. Air Force from 1966 to 1970,
and is a graduate of Albright College.
APPENDIX C 99
Wellesley College. Her work ranges from system building for afterschool
and youth development to professional development and creating evalu-
ation systems. She currently serves as one of the technical assistance pro-
viders for the Wallace Foundations Next Generation Afterschool System
Building Initiative. She is the principal investigator for the Robert Bowne
Foundation Afterschool Matters Initiative and is project director for the
Massachusetts Department of Elementary and Secondary Education techni-
cal assistance and training initiative for 21st Century Community Learning
Center grant recipients. She also serves as senior project advisor on NIOSTs
Afterschool Program Assessment System. She is a founding member of the
Health Out-of-School Time Coalition and was a national board member of
the American Camp Association and is a past co-chair of the Next Genera-
tion Youth Work Coalition. She received her B.S. from the University of
Massachusetts Amherst, and her M.Ed. from the Lesley College Graduate
School of Education.
The P
ennsylvania State University and received his Ph.D. from Howard
University.
Stephanie Jones (Committee Member) is the Marie and Max Kargman asso-
ciate professor in human development and urban education at the Harvard
Graduate School of Education. Her research focuses on the longitudinal
effects of poverty and exposure to violence on social, emotional, and behav-
ioral development from early childhood through adolescence. Much of her
recent work has focused on exploring noncognitive factors across the devel
opmental spectrum, with an emphasis on conducting rigorous scientific
research while also creating translational and applied products for the early
and middle childhood practitioner and policy communities. Jones serves on
numerous national advisory boards and expert consultant groups related to
social-emotional development and child and family anti-poverty policies.
She has experience conducting large-scale literature reviews; creating multi
disciplinary, integrative conceptual frameworks; and translating research
into accessible content. She received the Grawemeyer Award in Education
for her work with Zigler and Walter Gilliam on A Vision for Universal
Preschool Education, and the Joseph E. Zins Early-Career Distinguished
Contribution Award for Action Research in Social and Emotional Learn-
ing, from the Collaborative for Academic, Social, and Emotional Learning.
She holds a doctorate in developmental psychology from Yale University.
Mary Keller (Presenter) serves as the president and CEO of the M ilitary
Child Education Coalition. She has been the organizations executive leader
since 1998 and was one of its founders. She served as a teacher and admin
istrator in several Texas school districts for more than 21 years. She served
for 8 years as assistant superintendent and area superintendent for educa-
tion services for the Killeen Independent School District, which serves more
than 20,000 military connected children and the nations largest military
installation, Fort Hood. She holds professional certifications in teaching
elementary education, as well as history, supervision, mid-management,
and superintendency. She also holds a mediation certification from the
Texas Bar Association. She graduated from Wayland Baptist University
with a bachelors degree in elementary education and a masters degree in
education with a specialization in curriculum and instruction. She earned
her doctorate in educational administration with a special emphasis in
supervision from Texas Tech University.
APPENDIX C 101
ogy, and has authored more than 200 refereed publications and presenta-
tions. He has received multiple awards for his work, including the Farleigh
Dickinsons Distinguished Faculty Award for Research and Scholarship and
the Society for Personality Assessments Martin Mayman Award for distin-
guished contribution to the literature in personality assessment.
Velma McBride Murry (Committee Member) is the Lois Autrey Betts chair
in education and human development and professor of human and organi
zational development in Peabody College at Vanderbilt University. Her
APPENDIX C 103
APPENDIX C 105
Appendix D
What is Character?
What Works in Developing Character?
Investing in Implementation and Evaluation
Measuring Character
Please feel free to use these worksheets for note-taking during the workshop
and as a guide for discussion during the breakout sessions.
107
How does Larry Nuccis presentation about what character is and the dis-
cussion that followed influence your thinking about your programs objec-
tives, or the role of programs like yours in developing specific attributes?
Think about the full range of activities and practices used in your program
that may play a role in developing character, and how they may do that.
Begin by writing a list of the major activities of your program. Next, make
a list of the key outcomes your program hopes to address. Draw an arrow
connecting each of the activities to specific outcomes. What research can
you associate with those arrows? Which connections are supported by
research? Which are not?
[Use the table on the next page.]
APPENDIX D 109
4. What ideas about effective features and practices could help you meet
your objectives?
APPENDIX D 111
APPENDIX D 113
How does Bill Trochim and Jen Urbans presentation about advancing
evaluation of character building programs and the discussion that fol-
lowed influence your thinking about the role of your program and how
programs like yours can engage in evaluation practice and develop a culture
of evaluation?
APPENDIX D 115