Riccio, G. (2010). Integration of Leadership, Education, Training, and Self-Development. In: Riccio, G., Diedrich, F.
, & Cortes, M. (Eds.). An
Initiative in Outcomes-Based Training and Education: Implications for an Integrated Approach to Values-Based Requirements (Epilogue). Fort Meade, MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word frequency in text.] Epilogue 254 Asymmetric Warfare Group Epilogue. Integration of Leadership, Education, Training, and Self-Development
Gary Riccio The Wexford Group International
The purpose of this final chapter is to summarize the topics of discussion that, by the end of the AWGs scientific engagement in OBTE, were most urgent and vibrant with respect to implications for social and programmatic decision-making about Army training, education, and leader development.
Toward Values-Based Standards for Army Doctrinal Requirements
To date, the proponents and participants in OBTE have stressed that we should consider all the outcomes of specific learning events that are relevant to preparation of Soldiers for Full Spectrum Operations (FSO), not just the outcomes that are intended or explicit in a Program of Instruction or Training Support Package. This has led us to the conclusion that OBTE provides an overarching vision for training, education, and leader development that makes all these endeavors relevant to FSO and to operational commanders in theater (see Chapters 3, 4, and 5; see Figure 1). The principles and practices of OBTE follow, arguably in a deductive fashion, from this vision of relevance. While not especially surprising theoretically, at least in hindsight, this conclusion has important practical implications. Most importantly, it suggests that OBTE is both a manifestation and response to existing Army doctrine. The cascading implications are outlined below.
Figure 1. Triadic balance and equilibrium among intangible personal attributes and the associated scaffolding for capabilities of the human dimension in Full Spectrum Operations and competence in trans-extremis environments (e.g., transitions between strategically significant combat and noncombat operations). See Chapter 3, section 3.2.3.
255 Riccio
Asymmetric Warfare Group The current notion of OBTE as vision for learning and development suggests that we can approach the relationship between immediate learning objectives and long-term developmental outcomes from the opposite direction, in a sort of reverse planning from what the Army expects of its Soldiers coming out of programs of training and education. The intangibles such as confidence, initiative, and accountability help bridge the gap between behavior that is immediately observable and behavior that is meaningful and efficacious in FSO. But what are the ultimate outcomes? Our current belief is that OBTE does not suggest a fundamentally new set of concepts to be added to doctrine. Given the vision and the purpose of preparing for FSO, outcomes should be derived from doctrinally accepted concepts such as Army Values and Warrior Ethos or at least they should be traced to them.
One of the strengths of OBTE is that its implementation can be verified. OBTE thus can help translate existing doctrinal requirements such as Warrior Ethos and Army Values into practices that actually are observable and around which programs of instruction can be designed and developed. By showing that intangibles can be observed through their behavioral manifestations (a common proposition in the physical, biological, and behavioral sciences) and that measures can be developed for observable behavior, we also can claim that standards can be identified for such concepts. The logical conclusion of this line of argument is that a rigorous scientific approach to development, verification, and validation of OBTE provides a way to identify standards for value-based requirements in existing doctrine for which there currently are no measures or standards.
A more axiomatic formulation of this line of argument is:
(1) The most fundamental aspects of doctrine are the requirements that are explicit or implicit in Army Manuals, Regulations and associated documents. Concepts like Army Values and Warrior Ethos are among the most common requirements stated explicitly in Army doctrine.
(2) The Army is a standards-based organization but there currently are no doctrinally codified standards for some of the most important requirements in Army doctrine (e.g., as Army Values, Warrior Ethos).
(3) Verifiability and validity are two universally accepted criteria in capabilities development and quality assurance that all standards should address directly. Verifiability means that one can observe something tangible that relates directly to the standard (e.g., one can verify that one is doing what one says one is doing or should be doing). Validity refers to the relevance of the standard to the requirement (e.g., does training to the standard help one meet the requirement).
(4) The validity of all standards (e.g., current marksmanship qualification standards) is intangible with respect to some of the most important requirements in the Army (i.e., Army Values, Warrior Ethos). The relationship is inductive not deductive. Nevertheless, not all standards and inductive relationships are created equal. One should develop, refine, and utilize standards for which confidence in validity is greatest (i.e., for which evidence about validity has the most depth and breadth).
(5) To date, OBTE is the only approach to training, education, and leader development that has generated measurespotential standardsfor some of the most important requirements in the Army (e.g., Army Values, Warrior Ethos). While there is evidence for the validity of these potential standards, the more important point is that OBTE has shown that such standards can be developed in principle. Epilogue 256 Asymmetric Warfare Group
(6) Tasks, conditions, and standards are particular solutions to particular requirements. The most desirable solutions are ones that can evolve though refinement, upgrades, or other adaptations to a changing context for the associated requirements over the life cycle of the solution. Verification and validation fosters continued viability and relevance of a solution by providing feedback about what to sustain and what to improve.
(7) OBTE arguably is the only approach to training, education, and leader development that has generated a methodology for integrated verification and validation.
The application of OBTE to doctrine embodies a broader conceptualization of standards, one that connotes a quality of something that is attainable, not just standardization and evaluation with respect to norms. The implications about standards and their employment reveal another way that OBTE transcends particular instructional methods. Irrespective of the methods utilized in training and education, there is a need for further development of standards for values-based requirements in current Army doctrine. The validity of such standards cannot depend critically on any particular methodology even though, in particular situations or under particular conditions, some instructional methodologies may be more effective than others in achieving values-based standards.
Nested Standards and Quality Assurance
The elaboration and refinement of values-based standards provides critical guidance to instructors and their chain of command but it is not sufficient. More guidance is needed with respect to how such standards can be achieved. Without clarity about the means to the ends, recognition of gaps or shortfalls in achieving standards will not be actionable; there will not be a way to assure that achievement of standards will be sustained let alone to identify ways to improve instruction with respect to the achievement of standards. In this respect, the most important contribution of the scientific engagement in OBTE to date has been the development of formative measures for instructors (e.g., Chapter 2, Appendix A) and the scientifically based approach to verification and validation with which the measures are associated. The measures are the only way to verify that OBTE is being practiced as it has been defined and thus to validate that it can be effective. They provide a rigorous framework for the development of values-based standards that are achievable.
More generally, the most important realizations about OBTE are that there are requirements in Army doctrine for which solutions are undefined or unverifiable, that measurable standards can be developed for them, that values-based standards need not displace existing knowledge-based or skill-based standards, that the achievability of values-based standards can be verified with formative measures for instructors, and that achievement of values-based standards need not require additional resources for training and education. The logical implication of these realizations is that multiple standards of different kinds should be addressed in every learning event. Note that the implication is not that values-based standards are merely a higher level quantitatively on measures associated with existing standards. They are standards of a fundamentally different kind.
The formative measures developed for OBTE (Appendices A and B) provide a menu from which instructors, and other stakeholders with an interest in quality assurance, can select a manageable number of measures for planning, executing, and learning lessons from a particular learning event. None of these measures precludes assessment with respect to knowledge-based or skill- based standards, although they certainly can inform priorities and tradeoffs with respect to an exhaustive list of tasks, conditions and standards that are not practicable (Appendix C). The 257 Riccio
Asymmetric Warfare Group evidence from this investigation supports these claims, in essence, that multiple measures of different kind can be used in the same learning event and that values-based measures in particular can inform decision-making in planning and executing instruction (e.g., Chapters 8, 9, 13). This realization offers great promise for packaging training and education in more efficient ways (e.g., Perry & McEnery, 2009). OBTE, and specifically the associated formative measures, provide clues about how to link tasks to achieve multiple goals at the same time. Thus, in addition to showing that values-based standards dont impose an additional resource load, formative measures for OBTE may even show a path to achieving more with fewer resources.
It also is clear that there is much to be learned about the efficient and effective use of multiple measures. Importantly, given the foundation provided by this investigation, increasing clarity on such critical issues now is largely a matter of paying attention to the most important outcomes of training and education whether intended or not, focusing collective priorities in a program of instruction within a command intent that addresses the most important outcomes, and documenting progress (Chapter 11; Appendix C). It is noteworthy that the value of these actions, and the clarity they bring, is not limited to programs that adhere to the principles and practices of OBTE. The principles and practices of OBTE will, however, bring greater value insofar as they were developed pointedly to address values-based outcomes (Chapters 1-5).
Key constructs in OBTE are intangibles such as confidence, initiative and accountability. They are assumed to have causal potency with respect to the developmental outcomes that motivated OBTE. The trustworthiness of this assumption is revealed by the extensive dialectical reflection and analysis we conducted during measure development, field-based observation, and the development of grounded theory for the practical wisdom of Soldiers as leaders and instructors (e.g., Chapters 2-5). The intangibles emphasized by OBTE relate directly to values-based doctrinal requirements such Army Values and Warrior Ethos (see e.g., Chapter 3; Appendix D). The identification and promulgation of values-based standards will require additional attention to outcomes that are influenced by these intangibles (cf., Bandura, 1997; see Figure 1). There is a need to identify and refine the understanding of such outcomes as effects on students that can transfer to different situations including performance in Full Spectrum Operations (cf., Leibrecht, Wampler, & Pleban, 2009) if not to other aspects of a Soldiers life (see Chapter 4, section 4.3.2; Chapter 5, section 5.2).
We recommend that development of values-based standards consider the outcomes defined or refined locally in collaborative decision-making among the instructor cadre within the local commanders intent (Chapter 3, section 3.2.7; Chapter 14; Appendix C). Such outcomes and associated measures currently are in the early stages of decentralized development across various programs (Haskins, 2009; Perry & McEnery, 2009). Outcomes can be addressed systematically across programs and within a time scale that is short relative to the refinement of doctrine. Doctrine can assimilate this localized innovation and translate it into guidelines that stimulate and focus innovation rather than discourage it. Within such guidelines, instructors and their chains of command are in the best position to identify observable and achievable outcomes with the most relevance to stakeholders for their programs of instruction. Collaborative identification of outcomes is a means for instructors to develop and direct internal sources of motivation for good instruction. It is a means for the instructional organization to operate coherently with respect to the commanders intent and to achieve a command climate within which unnecessary unintended friction is readily identified and eliminated.
Epilogue 258 Asymmetric Warfare Group
Figure 2. Programmatic view of the development and employment of values-based standards as well as associated instructional principles, practices, and measures (i.e., OBTE). Solid lines represent direct programmatic influences. Dotted lines connote informal or indirect programmatic influences. Heavy solid lines connote direct psychological causes and effects of instruction on students. Compare with Figure 8 in Chapter 3.
To summarize key points about multiple standards:
Standards for values or ethos can be addressed in the same learning event as existing standards.
Tasks, conditions, and standards can be designed to allow multiple standards to be addressed in the same learning event, conditions permitting.
Addressing multiple standards in the same learning event suggests the potential to accomplish more with the same resources or less.
Standards should reflect outcomes that are developed in decentralized innovation across programs of instruction.
259 Riccio
Asymmetric Warfare Group Needs and Opportunities for Staff & Faculty Development
A Role for Science and Measurement
Several promising directions for instructor education in OBTE are outlined below. While the recommendations are based on established practices, they are nuanced in the context of OBTE. The nuances derive from the intent to be directive about process while not being directive about content (see e.g., Chapter 3, section 3.2.7; Chapter 14; Appendix C), to foster flexibility and adaptability in instructional planning and execution, and to strike a balance between the science of teaching and the art of teaching. The subtlety we seek was captured well by William James in his talks to teachers over a century ago:
The teachers of this country, one must say, have its future in their hands. The earnestness which they at present show in striving to enlighten and strengthen themselves is an index of the nations probabilities of advance in all ideal directions [p. 3] You make a great, a very great mistake, if you think that psychology, being the science of the mind's laws, is something from which you can deduce definite programs and schemes and methods of instruction for immediate school-room use. Psychology is a science, and teaching is an art; and sciences never generate arts directly out of themselves. An intermediate inventive mind must make that application, by using its originality [p. 7] A science only lays down lines within which the rules of the art must fall, laws which the follower of the art must not transgress; but what particular thing he shall positively do within those lines is left exclusively to his own genius [p. 8]... many diverse methods of teaching may well agree with psychological laws [p. 9] To know psychology, therefore, is absolutely no guarantee that we shall be good teachers. To advance that result we must have an additional endowment altogether, a happy tact and ingenuity to tell us what definite things to say and do when that pupil is before us. That ingenuity in meeting and pursuing the pupil, that tact for the concrete situation, though they are the alpha and omega of the teachers art, are things to which psychology cannot help us in the least [p. 9] We know in advance, if we are psychologists, that certain methods will be wrong, so our psychology saves us from mistakes. It makes us, moreover, more clear as to what we are about. We gain confidence in respect to any method which we are using as soon as we believe it has theory as well as practice at its back. [p. 11] (James, 1899/1907).
A vast amount of pedagogically relevant research has been conducted in the century since Jamess remarks but his advice is still relevant. Our advantage today is that psychology and other academic disciplines are in a better position to help instructors become more keenly aware of where certain methods will be wrong, to save us from mistakes, and to give us confidence in respect to any method which we are using as soon as we believe it has theory as well as practice at its back. We were mindful of Jamess advice from the beginning of the investigation, and it thoroughly infused our program of research. In general, through our interactions with the progenitors and stakeholders of OBTE, we diligently sought to identify the tact and ingenuity to tell us what definite things to say and do when that pupil is before us. Concurrently, we identified relevant science and scholarship that lays down lines within which the rules of the art must fall, laws which the follower of the art must not transgress (see e.g., Chapters 2 and 3).
Without something like the formative measures for instructors developed in this investigation, for example, there is no principled basis for instructor education with respect to values-based Epilogue 260 Asymmetric Warfare Group requirements in current Army doctrine. The student measures that were added toward the end of the scientific engagement (Chapter 12; Appendix B) also can provide formative feedback to instructors and to students about the efficacy of instructor behavior. They are measures that show whether the instructor is having some effect on the student and what some of those immediate effects are. However important, they are merely indicators (i.e., efficient causes) that outcomes are likely to be influenced by instructors, especially if high scores are obtained on the instructor measures (see Chapters 8 and 9). These can be dependent variables in a study of OBTE but they should not be confused with outcomes. With regard to outcomes emphasized by OBTE, they are intervening variables. The grounded theory outlined in Chapters 3-5 provides a foundation for further research into measures that provide a more complete picture of the impact instructors and their programs of instruction can have on individual learning and development with respect to values-based outcomes.
Toward Best Practices in Instructor Education
OBTE illuminates both needs and opportunities for development of staff and faculty associated with Army training, education, and leader development. It is important to note that none of these actionable implications are uniquely required by OBTE. They apply to any approach to training, education, and leader development, in principle, and they apply to current practices in these areas. The need for increased resources often is associated with OBTE merely because practitioners have a keener awareness about the best use of resources. They have greater motivation to achieve more or at least they have a more coherently directed motivation. The insights of OBTE may simply stimulate reconsideration about the distribution of resources. It may also reveal that an increase in resources devoted to training and education, such as instructor education, can have a disproportionately high increase in return on the investment. Again, this is not to be confused with a requirement of OBTE for more resources.
Field-based training. The AWGs Combat Applications Training Course (CATC) is an effective way to familiarize instructors with OBTE (see Chapter 6). Such opportunities for experiential learning are an important element of instructor education but CATC is not sufficient to ensure that instructors will understand the general principles of the new approach. Instructors should be familiarized with the application of OBTE to different skill sets or knowledge sets (see Chapter 11). We recommend use of formative measures (see e.g., Chapter 2) in such courses to assure quality and for collaborative reflection with participants in the course. The measures would help student-instructors appreciate the ways in which the general principles and practices of OBTE are manifested in particular instructional strategies and events. They would help see beyond the particulars of the learning experience to reveal the deep structure of learning and instruction.
Seminars and workshops. The AWG has provided seminars on a regular basis to stakeholders in OBTE (AWG, 2008a,b, 2009). These events have been very useful in helping instructors and their chain of command come to understand the vision, purpose, principles, and intended outcomes of OBTE even though the events are only several hours in duration. We prefer a workshop format with two half-day periods on consecutive days given that discussion about OBTE generally simulates subsequent reflection with noteworthy personal implications. We expect that workshops will be more successful when they are coupled with experiential learning (e.g., CATC). This assumption is consistent with our informal observations during this investigation.
We have recommended that workshops include multiple sessions in which the topic of discussion is the experience of instructors (Sidman, Riccio, Semmens, et al., 2009). Toward that end, it would be valuable to have a corpus of narrative vignettes that provide concrete examples of an 261 Riccio
Asymmetric Warfare Group instructors thinking and behavior. The vignettes could describe instructional events at a level of detail that provides enough substance and enough left unstated to stimulate meaningful collaborative reflection on a time scale commensurate with the time scale of the events described in the vignettes. The formative measures for OBTE (e.g., Chapter 2) provide an excellent framework both for generating such a corpus and for discussing the vignettes. In a sense, instructional vignettes can provide a way for the student-instructor to get inside the head of a more experienced instructor as conditions are observed, as in-stride decisions are made, and as consequences unfold. We believe this would help demystify good instruction. It also would have the added value of showing student-instructors how to use formative measures for quality assurance and self-development.
Collaborative decision-making. We have stressed the importance of participation of instructional cadre and their chain of command in making decisions about outcomes and associated measures of interest in their own programs (Chapter 3; section 3.2.7; Chapters 7 and 14; Appendix C). These collaborative discussions have the added value of helping individual instructors develop a broader and deeper understanding of OBTE. It is another way of learning from peers and contributing to that learning. It provides more direct connections between general theory and immediate practice.
Multimedia products. An advantage of a field-based course is that a student-instructor can see examples of good instruction. One of the limitations of this experiential learning is that it can be difficult to differentiate the essential from the incidental, to appreciate the general principles and practices in the particulars in which they are manifested. Exposure to a variety of instructors and applications of OBTE can be a powerful and effective method to help students generalize and to learn more deeply. Video records would be a simple, low-cost, and highly accessible method for student-instructors to be exposed and re-exposed, on demand, to a variety of instantiations of OBTE. Moreover, established methods in multimedia instruction could blend aspects of field- based training and seminars. We have developed multimedia materials for instructor education that utilized formative measures for values-based requirements to guide viewers through video segments of instructional events (Bruny, Riccio, Sidman, Darowski, & Diedrich, 2006; cf., Straus, Shanley, Burns, Waite, & Crowley, 2009).
There is added value to multimedia methods of instructor education that address a recognized need. Student-instructors and their leadership sometimes express skepticism that they are capable of the kind or level of instruction demonstrated by master instructors or by individuals with extraordinary credentials as Soldiers. While OBTE or any instruction in the Army can benefit from extraordinary experience and expertise in Soldiering, we believe that OBTE does not require such levels of mastery. Common instructors have the potential to improve their instruction with respect to the principles and practices of OBTE (e.g., Chapters 8 and 9). Nevertheless, it would be valuable for student-instructors to be exposed to instructors like me in situations like mine. This need not be limited to instructors. Multimedia materials also can highlight the role of an effective Brigade or Battalion commander, a tipping-point First Sergeant or Company commander, or other influential leaders.
Distributed collaboration. Collaborative decision-making and peer-to-peer sharing of lessons learned need not be limited to forums in which all participants are physically co-located (Chapter 11). There will be increasing opportunities for web-based social networking for the foreseeable future. Such capabilities should be leveraged for formal or ad hoc instructor education. One of the current needs in this context is finding outside experts who can help relative novices find the right information in the right amount and in the right amount of time. More generally, there is a need to help individuals share their respective expertise in the context of common or convergent interests. Epilogue 262 Asymmetric Warfare Group Internet portals that facilitate such decentralized ad hoc task organization are a promising potential solution to this need (Mikroyannidis, 2007; Riccio, Lerario, et al., 2006). Web-based utilities that enable user generated content and its distribution are transforming use of the web as well as the perception of what the web affords. This can lower barriers to entry into communities of practice and thus extending the opportunities to engage in and benefit from volunteerism (McKee, 2007; Putnam, 2001; Thoits, & Hewitt, 2001). \ Textbooks, pamphlets, and pocket guides. Textbooks are how novice instructors outside the military teach with little or no background in teaching and even sometimes with little background in the subject matter. One generally can use a text however one wants but at least it provides a framework and reference for one's local and momentary initiative. If the textbooks dont help instructors do this, they are not chosen or used. In any case, the authors get useful feedback about the value of their contributions. This market-based solution does not necessarily require monetary gain by the authors and may not even require monetary transactions. Other motives and transactions can connect volunteers with people who value their special expertise. Recognition, inclusion, and opportunities for impact can be powerful motivators (Maslow, 1943, 1968). Content generated by volunteers would require review and perhaps editing by TRADOC but this demands fewer resources than generating content. With regard to content for instructor education, reviewing and editing would be simplified significantly by the formative measures developed in this investigation (see Chapters 2, 7, 12).
In the context of OBTE, textbooks need not and should not tell instructors what to do. They should provide guidance about how to become a more effective instructor. We believe that an excellent source of guidance would be a corpus of narrative vignettes that provide concrete examples of an instructors thinking and behavior as suggested above in the context of workshops for instructor education. In any case, formative measures for instructor behavior are a valuable framework within which to seek, provide, and interpret guidance about instruction. Given that such guidance is not a script for everything an instructor does in a course, it can be concise. Mnemonics also can be produced on pocket cards for instructors who have been familiarized with the guidance represented in the cards. Pocket cards reflect the value of guidance that can be accessed in situ. This is especially important for novices who may not have fully assimilated lessons learned or retained knowledge of best practices or who may be overloaded during an instructional event (Freeman, Jason, Aten, et al., 2008).
Portable electronic aids. Elaborate frameworks and context-dependent guidance can be extremely valuable in situ. Hand-held electronic devices are becoming increasingly common aids to performance in everyday tasks as well as on the job. As such, they are a natural extension to pocket cards that are at once ubiquitous and excessive in the Institutional Army and the Operational Army. Electronic aids are interactive, thus, they allow the presentation of information to be as simple or as complex as needed. Clever design of menus and the underlying structure also can help a user find information without knowing exactly what they are looking for. More generally, use of an electronic aid and the information it provides can be context dependent.
Moreover, read-write capabilities enable hand-held electronic devices also to replace the green book that is a common tool for collecting notes and observations in the field, especially during instructional events. The capability to record noteworthy events within a framework of formative measures can facilitate both instruction and learning. It can greatly facilitate collaborative reflection either in formal After-Action Reviews or in ad hoc groups after a noteworthy event (see Chapter 3). Such devices also offer the added value of facilitating the capture and sharing of lessons learned with peers and stakeholders (see e.g., Riccio, dEchert, Lerario, et al., 2006).
263 Riccio
Asymmetric Warfare Group Critical Considerations for Further Scientific Investigation
The Necessity of Long-Term Studies
In our view, all the debate about OBTE reduces to lack of agreement about the impact of training and education, in general, irrespective of approach. By way of analogy, any competition that would pit OBTE against some other approach would be impossible to initiate until all parties agree on the rules of the game. The various parties in the debate about OBTE are not even close to being on the same page about what outcomes matter. In fact, the use of the term outcomes across the Army, and more broadly in education, has become so muddled it may be wise to consider dropping the term altogether (see Chapter 3). In any case, in the terminology of empirical science, the most important and difficult questions about OBTE involve the choice of dependent variables. If we can achieve some agreement on what effects to measure, then we can tackle the tough decisions about independent variables (i.e., interventions, treatments) or comparisons (e.g., demographics and characteristics of samples, groups, situations).
OBTE emphasizes long-term developmental outcomes consistent with values-based requirements in existing Army doctrine. Scientific inquiry can play an important role in identifying behavioral manifestations (i.e., outcomes) of the values developed in Soldiers as well as antecedents that reflect progress in the development of such values. Short-term investigations such the ones reported in this manuscript are valuable but they are not sufficient for overarching programmatic decisions about OBTE. The value of the empirical work summarized herein is in revealing proximate effects of any approach to training and education and, more importantly, in verifying the implementation of any approach (Chapter 1). It cannot, by definition, reveal long-term effects such as inculcation of values described in Army doctrine. Long-term studies are warranted if not required if science is to help the Army meet its existing requirements. Moreover, theory development is necessary to explore connections between short-term and long-term effects (see e.g., Chapters 3-5). Without theory, it is difficult to take any empirical investigation seriously, and it certainly is impossible to draw any actionable conclusions based on the evidence. The broader and deeper the theory, the more confidence one can be about interpretations of the evidence, and the more responsible one can be in making recommendations based on the evidence. The seriousness of the tasks currently faced by Soldiers in FSO deserves theory that is commensurate in depth and breadth.
OBTE claims that, when there is singular focus on a task and prescribed conditions associated exclusively with a knowledge-based or skill-based standard, there is an unintended reliance on short-term memorization and conditioning. This may result in a reasonable level of performance in the learning event but our claim is that it also results in an unintended negative impact on long- term outcomes. In particular, it can be counterproductive with respect to attributes and values necessary to perform in ambiguous situations or across the variations implicit in Full Spectrum Operations (FSO).
OBTE claims that, when leadership at all levels seeks to create a command climate that strongly encourages the development of values and attributes emphasized by OBTE, the instructors will naturally begin to identify opportunities to achieve the knowledge-based or skill-based standard established for performance while also developing a Soldier with respect to long-term outcomes. Further, we argue that this will lead to the discovery of new and better ways to instruct with respect to short-term objectives and long-term outcomes within current resource constraints; that is, it will lead to a more efficient use of resources. Such adaptability and ingenuity is exactly what is needed in FSO.
Epilogue 264 Asymmetric Warfare Group In essence, OBTE claims that the Army is already conducting outcomes based training and education, in that there is always long-term outcomes from every event or encounter with a student. A singular focus on knowledge-based or skill-based standards, however, can lead to negative outcomes. These are empirical questions, and this is where science can help. These issues suggest that further research would do well to include social psychological and even econometric perspectives (see Chapters 4 and 5) more deeply than we were able to address in the current investigation which initially drew mostly on industrial-organizational psychology and the experimental psychology associated with human perception and performance, cognition and learning, and human development (see Chapter 3). Further research should be re-conceptualized at a higher level in terms of inter-temporal risk-benefit or cost-benefit relationships, for example:
Benefit: What is the impact of any program of training and education on outcomes that matter for mission effectiveness in Full Spectrum Operations?
Risk: Does OBTE have a positive or negative effect with respect to near-term learning objectives in a particular program of instruction?
Risk-Benefit Analysis: Does impact of OBTE on outcomes justify additional risk, if any, to near-term learning objectives?
Cost: Does OBTE require additional costs for materiel or non-materiel resources or redistribution of such resources to maintain overall cost?
Cost-Benefit Analysis: Does impact of OBTE on outcome justify additional cost or redistribution of costs, if any, of its adoption?
Note that risk-benefit and cost-benefit analyses are relevant only if there is a tradeoff, that is, if outcomes come at a risk relative to near-term learning objectives or at cost in additional resources used. With improved instructional design informed by experience with OBTE, we believe that there will be no tradeoff. This seems to be the case at sites where instructor cadre and their leadership have collaboratively re-designed their courses around OBTE (Haskins, 2009; Perry & McEnery, 2009; Appendix C).
False Dichotomy of Objective-Subjective
Measures and standards that relate to long-term developmental outcomes will look different from measures typically used to assess performance with respect to knowledge-based and skill-based standards. There is some concern about subjectivity in measures associated with values-based standards and, in particular, there are some assumptions about unreliability relative to objective measures associated with more common standards. As a synonym for reliability and validity, objectivity is a worthy pursuit and one that is convergent with rigorous subjectivity, especially inter-subjectivity (Bridge, 2000; Dienes & Perner, 1999; Neisser & Fivush, 1994; Reed, 1996; Searle (1997). The dichotomy is largely an illusion, however, for a variety of reasons.
Consider standards being applied in training today that are assumed to be objective. For any standard, there are numerous ways to get to that standard that are not specified or precluded in the associated regulations. Some of the unspecified ways of approaching a standard would be interpreted as cheating, others are not so clear because of judgment on the spot when conditions are not as assumed in the standard. In either case, the result depends critically on subjectivity. In addition to this practical constraint on objectivity, there may be legitimate questions about the putative validity of the standard with respect to the operational task to which it ostensibly is 265 Riccio
Asymmetric Warfare Group relevant. Potential violations of the assumption about validity generally are an even more serious challenge to the common connotations of objectivity. The only way out of this conundrum is to trust instructors and evaluators in using judgment (subjectivity), to establish a command climate that motivates and fosters trustworthiness (shared values), to provide instructors and evaluators with instruments and guidelines that are trustworthy (rigorous) without precluding judgment, and to engage continually in meaningful collaborative reflection (inter-subjective validity).
When focused on nave objectivity (assessments that could be done by a novice), measures of performance can pervert the purpose of training and education. Without the currency and relevance (that novices are incapable of considering) mere verifiability and repeatability are insufficient to assure development of the Soldier. Novice assessments tend to focus on specific aspects of the nominal training and education process rather than on the various outcomes, especially intangible outcomes, that the process influences. Novices can be expected to depend upon these factors and to operate so as to maximize the arbitrarily observable aspects of their instruction and minimize the role of their own expertise as a Soldier and instructor. On the other hand, if instructors focus on intangible outcomes, their experience as a Soldier and instructor is a critical foundation for reliable assessment of good training. Leaders are familiar with assessing intangible attributes. The systems for evaluating Officers and Non-Commissioned Officers depend on the ability of experienced raters to make judgments about competencies, skills, values and other qualitative characteristics of an individual (AWG, 2009). This is no less valid for assessing training and education.
The behavioral and social sciences have developed a plethora of methods that attempt to strike a balance between objectivity and subjectivity, that reflect both considerations rather than an assumption that one has to choose between them, and thus that provide sophisticated and trustworthy perspectives on verifiability and validity (Chapters 4, 5, and 11). There is no reason for programmatic decision makers to ruminate over these problems of measurement or, worse, to make suboptimal and even counterproductive decisions because of an assumption that the problems are insolvable. Science provides plenty of options for moving forward, especially when it is embedded in a broader scholarly context (Chapters 3-5; see also, Flyvbjerg, 2001; Godfrey- Smith, 2003; Schrim & Caterino, 2006).
Clarity About What Is Evaluated
We have developed measures that reflect best practices of OBTE and, most importantly, are verifiable in practice. Without this, there is no way of knowing whether, or the extent to which, OBTE actually is implemented; and without such verifiability, investigating the results in a program of instruction is meaningless. No scientific conclusions can be drawn nor can any improvements be suggested without verification of instructional practices. Documents describing a program of instruction, a training support package, or any other kind of script for a learning event do not satisfy this criterion. Measures are required so that scientists and evaluators can verify what is occurring on the ground at the same time measures of effectiveness are being applied and data on learning are being collected.
The focus of OBTE is on the instructor because instructors are the means through which the organization has influence on the students. Measurement of instructor behavior is necessary for this reason and, while it may be sufficient for scientific conclusions, it may not be sufficient to come to conclusions that are actionable operationally. The reason is that there are organizational factors that influence what an instructor can do (see Chapters 14 and 15; Appendix C). There is a need to understand best practices within an instructional organization, including but not limited to the command climate, with a level of theoretical grounding and practical detail commensurate Epilogue 266 Asymmetric Warfare Group with the formative measures we have developed for instructor behavior. The thinking that scientists would do to figure out what to measure within an organization is the very same thinking that a programmatic stakeholder would have to do to reexamine and reprioritize activities and objectives within a program. A provocative implication is that it would be useful to create a forum in which scientists, leaders in programs of instruction, and active duty commanders or their representatives could engage in collaborative reflection about the principles, practices, and desired outcomes of training and education. The collaboration in such a forum could look very much like the COMPASS process used to develop formative measures for OBTE (see e.g., Chapter 2).
The return on the investment in science to support programmatic decision-making is greatest if the scientific assessment yields information that either can improve the capability or refine the requirements that motivated the capability. One of the most important contributions of the scientific inquiry into OBTE to date is to argue for a reorientation toward the common competencies and attributes of Soldiers that matter to active duty units involved in Full Spectrum Operations, to strive for a fuller and actionable interpretation of requirements for training and education, and to close any associated gap between the Institutional Army and Operational Army. Another important contribution is to show how best practices of instruction can become more readily identifiable based on this reorientation. Together, this allows for systematic scientific inquiry and a broader scholarly dialog among scientists and nonscientists that can have the following results:
Stimulate debate and facilitate decision making about instructional methodologies Identify and facilitate a coherent approach to training, education, and leader development Provide framework for curriculum-level design and development Provide framework for continuous and rigorous quality improvement
Next Steps
Further research should vigorously pursue development of (a) standards for the values-based outcomes of interest to OBTE and Army doctrine; (b) measures that relate to various aspects and stages of individual progress with respect to values-based outcomes; (c) theory and empirical methodologies that reveal inter-temporal relationships between immediate observable effects and long-term outcomes; and (d) an approach to verification and validation that brings quality assurance into a tight correspondence with staff and faculty development within a comprehensive approach to program evaluation.
This broad program of research would not be possible without the grounded theory we developed over the course of this investigation (see e.g., Chapters 3, 4 and 5). These scientific underpinnings are just a beginning for serious scholarship into OBTE and, more generally, into the critical interrelationships among training, education, self-development, and leader development. At the same time, this foundation is sufficient to initiate empirical research and to being to weave together otherwise disparate sources of empirical evidence needed to support programmatic decision-making in the institutional Army (cf., Flyvbjerg, 2001; Godfrey-Smith, 2003; Schrim & Caterino, 2006; see also Chapter 11, section 11.2.3).
Figure 3 shows ways in which the scientific underpinnings for OBTE can guide the development of values-based standards that can be verified and validated. This should be viewed as a road map. The specific measures have yet to be identified and applied. It is important to note, however, that these scientific underpinnings for OBTE are sources of measures that have been scientifically validated and employed to come to a deeper understanding of situated behavior, situated 267 Riccio
Asymmetric Warfare Group experience, and situated meaning. That is, they preclude having to start from scratch and develop new measures for all the important activities, decisions, and processes that are critical in OBTE and, more generally, in the institutional Army. Our collaborative inquiry into the practical wisdom as instructors and leaders has given us a good start. The integrative work should be continued so that programmatic decision-makers can have the information they need to have the greatest impact on preparation of Soldiers for FSO (see Chapters 11 and 15).
Figure 3. Sources of potential measures to support analysis, design, development, implementation and evaluation of instruction that reflects a programmatic view of the development and employment of values-based standards. Compare with Figure 2 in this Epilogue and with Figure 8 in Chapter 3. Epilogue 268 Asymmetric Warfare Group References
Asymmetric Warfare Group (2009, March). US Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education, Applied Physics Laboratory, Johns Hopkins University, Laurel MD. Bandura, A (1997). Self-efficacy: The exercise of control. New York, NY: Freeman and Company. Bridge, G. (2000). Rationality, ethics, and space: on situated universalism and the self-interested acknowledgement of difference. Environment and Planning D: Society and Space 2000, 18, pages 519-535. Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006). Enhancing warrior ethos in initial entry training. Proceedings of the 50th Annual Meeting of the Human Factors and Ergonomics Society, San Francisco, CA. Darwin, M. (2008a). Asymmetric Warfare Group combat applications training course (CATC) senior leader discussion (Briefing, January 23, 2008). Fort Meade, MD: Asymmetric Warfare Group. Darwin, M. (2008b). Outcomes-based training and education: fostering adaptability in full spectrum operations (Briefing, December 2008). Fort Meade, MD: Asymmetric Warfare Group. Dienes, Z. & Perner, J. (1999) A theory of implicit and explicit knowledge. Behavioral and Brain Sciences 22(5), 735-808 Freeman, J., Jason, J., Aten, T., Diedrich, F., Cooke, N., Winner, J., Rowe. L., & Riccio, G. (2008). Shared Interpretation of Commander's Intent (SICI). Final Report to the Army Research Institute for the Behavior and Social Sciences, contract number W74V8H-06-C- 0004. Flyvbjerg, B. (2001). Making social science matter: Why social inquiry fails and how it can succeed again. Cambridge, UK: Cambridge University. Godfrey-Smith, P. (2003). Theory and reality. Chicago, IL: University of Chicago Press. Haskins, C. (2009, March). Development of outcomes based training. Presentation at the US Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education, Applied Physics Laboratory, Johns Hopkins University, Laurel MD. James, W. (1907). Talks to teachers on psychology: and to students on some of life's ideals. New York, NY: Henry Holt and Company. (Original work published 1899) Leibrecht, B., Wampler, R., & Pleban, R. (2009). Methodology for evaluating transfer of learning from the U.S. Armys advanced leaders course. Research Product 2009-05. Washington, DC: U.S. Army Research Institute for the Behavioral and Social Sciences. Maslow, A. (1943). A theory of human motivation. Psychological Review, 50, 370-396. Maslow, A. (1968). Toward a psychology of being. New York: Van Nostrand. McKee, J. (2007). The new breed: Understanding and equipping the 21st century volunteer. Loveland, CO: Group Publishing.Mikroyannidis, A. (2007). Toward a social semantic web. Computer, November 2007, 113-115. Los Alamitos, CA: IEEE Computer Society. Neisser, U., & Fivush, R. (Ed.) (1994). The remembering self: construction and accuracy in the self narrative. Cambridge, UK: Cambridge University Press. Perry, R. & McEnery, K. (2009). Army reconnaissance course: Defining the aim point for reconnaissance leader training. Armor, July-August, 14-20. Putnam, R. (2001). Bowling alone: The collapse and revival of American community. New York: Simon & Schuster. Reed, E. (1996). The necessity of experience. New Haven: Yale University. Riccio, G., dEchert, B.C., Lerario, M., Pound, D., Bruny, T., & Diedrich, F. (2006). Enhancing Joint Task Force Cognitive Leadership Skills. Report to the Army Research Institute for the 269 Riccio
Asymmetric Warfare Group Behavioral and Social Sciences, contract number Army contract no. W74V8H-06-P-0186. Vienna, VA: The Wexford Group International. Riccio, G., Lerario, M., Cornell dEchert, B., Pound, D., Bruny, T., & Diedrich, F. (2006). Training a Joint and expeditionary mindset. Report to the Army Research Institute for the Behavioral and Social Sciences, contract number W74V8H-06-P-0189. Vienna, VA: The Wexford Group International. Schrim, S. & Caterino, B. (2006). Making political science matter: debating knowledge, research, and method. New York: New York University. Searle, J. (1997). Construction of social reality. New York: Free Press. Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Diedrich, F. (2009). Reshaping Army institutional training: Current training. Final Report to the Army Research Institute for the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO 0010. Straus, S., Shanley, M., Burns, R., Waite, A., & Crowley, J. (2009). Improving the Armys assessment of interactive multimedia instruction courseware. Santa Monica, CA: RAND. Thoits, P.A., & Hewitt, L.N. (2001). Volunteer work and well-being. Journal of Health and Social Behavior, 42, 115131.
ix Riccio, Diedrich, & Cortes (Eds.) Asymmetric Warfare Group TABLE OF CONTENTS . page
Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education.......1 Historicity of our Research on OBTE..........................................................................................1 The Approach and Lessons Learned from the Research..............................................................3 Documentation of the Research ...................................................................................................4
Section I. Development of Stakeholder Requirements for OBTE..............................................6
Chapter 1. Preparation for Full Spectrum Operations ......................................................................7 1.1 Requirements of Full Spectrum Operations...........................................................................8 1.2 Outcomes-Based Training and Education (OBTE)..............................................................10 1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................11 1.2.2 OBTE as a Multifaceted Instructional System .............................................................12 1.3 An Appraisal of Instruction with Respect to OBTE ............................................................13 1.3.1 A Systems Engineering Framework for Integration and Development of OBTE........13 1.3.2 Preparation for Validation and Verification .................................................................14 1.4 References ............................................................................................................................17
Chapter 2. Formative Measures for Instructors ..............................................................................20 2.1 Development of Formative Measures ..................................................................................20 2.1.1 The COMPASS Methodology......................................................................................20 2.1.2 Development of Measures for OBTE...........................................................................21 2.2 Description of Formative Measures .....................................................................................21 2.2.1 Results of the COMPASS Process................................................................................21 2.2.2 Elaboration on the Description of Measures.................................................................23 2.3 OBTE Performance Measures: Planning for Training.........................................................23 2.3.1 Define Outcomes ..........................................................................................................23 2.3.2 Create a Positive Learning Environment ......................................................................25 2.3.3 Create the Parameters of Learning................................................................................27 2.4 OBTE Performance Indicators: Training Execution............................................................28 2.4.1 Communicate the Parameters of Learning....................................................................28 2.4.2 Training Emphasizes Broad Combat or Mission Success ............................................29 2.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................31 2.4.4 Facilitates Learning of Concepts ..................................................................................32 2.4.5 Creates a positive learning environment.......................................................................34 2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation................................36 2.4.7 Uses scenarios to facilitate learning..............................................................................38 2.4.8 Instructors exhibit intangible attributes in own actions ................................................40 2.4.9 Hotwashes and Mini-AAR............................................................................................42 2.5 Uses of the Measures ...........................................................................................................43 2.5.1 Formative Measures for Instructors..............................................................................44 2.5.2 Quality Assurance and Instructor Education ................................................................44 2.5.3 Continuous Improvement of Assessments....................................................................45 2.5.4 Program Evaluation and Organizational Change..........................................................46 2.6 References ............................................................................................................................46
Table of Contents x Asymmetric Warfare Group Chapter 3. Principles and Practices of Outcomes Based Training & Education............................50 3.1 Multifaceted Inquiry.............................................................................................................50 3.1.1 Interaction with Progenitors of OBTE..........................................................................51 3.1.2 AWG Documents on OBTE .........................................................................................52 3.1.3 Collaborative Reflection on Participant Observation in CATC ...................................52 3.1.4 Interaction with Stakeholders .......................................................................................53 3.2 Essential Characteristics of OBTE.......................................................................................53 3.2.1 The Meaning of Developmental is a Critical Difference..............................................53 3.2.2 The Definition of Outcomes is a Critical Difference....................................................56 3.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....58 3.2.4 The Meaning of Experience is a Critical Difference ....................................................61 3.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................62 3.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................63 3.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65 3.3 Toward a Grounded Theory for OBTE................................................................................66 3.3.1 Need for an Integrated Interdisciplinary Framework ...................................................66 3.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67 3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................68 3.4.1 Leadership and Enculturation of Soldiers.....................................................................68 3.4.2 Robust and Adaptable Plan...........................................................................................70 3.4.3 Instructors as Role Models ...........................................................................................70 3.4.4 Collaborative Identification of Outcomes and Measures .............................................71 3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................71 3.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........72 3.5.2 Task Relevance of Planned Instructional Events..........................................................72 3.5.3 Reveal Operational Relevance of Training...................................................................73 3.5.4 Incorporate Stress into Instructional Events .................................................................73 3.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74 3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment ...................74 3.6.1 Collaborative Reflection and Problem Solving ............................................................75 3.6.2 Communication.............................................................................................................75 3.6.3 Nature and Extent of Guidance.....................................................................................76 3.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76 3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................77 3.7.1 Soldier Motivation and Development of Intangibles....................................................77 3.7.2 Plan for Development of the Individual .......................................................................78 3.7.3 Get Students to Take Ownership ..................................................................................78 3.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79 3.8 References ............................................................................................................................79
Chapter 4. Grounded Theory for Values-Based Training & Education.........................................86 4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................86 4.1.1 Fundamental Units of Analysis.....................................................................................87 4.1.2 Nested Time Scales and Adaptability...........................................................................88 4.1.3 Adaptability and Ambiguity .........................................................................................90 4.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92 4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93 4.2.1 Ecological Psychology..................................................................................................93 4.2.2 Self-Efficacy Theory.....................................................................................................97 4.2.3 Positive psychology ......................................................................................................98 4.3 A More Integrated Scientific Infrastructure.......................................................................101 xi Riccio, Diedrich, & Cortes (Eds.) Asymmetric Warfare Group 4.3.1 Self Determination Theory .........................................................................................101 4.3.2 Situated Learning Theory ...........................................................................................103 4.3.3 Existential Psychology................................................................................................105 4.4 Building on the Scientific Infrastructure for OBTE...........................................................109 4.4.1 Triadic Frameworks....................................................................................................109 4.4.2 Further Development ..................................................................................................112 4.5 References ..........................................................................................................................112
Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118 5.1 The Nested Self ..................................................................................................................118 5.1.1 An Alternative to Individual versus Collective ..........................................................118 5.1.2 Cognition and Reality .................................................................................................119 5.2 Conscious Experience and the Dynamics of Thinking ......................................................122 5.3 Emotion, Information, and Engagement ............................................................................125 5.3.1 Ecological Perspective on Emotion ............................................................................125 5.3.2 Emotion as Engagement .............................................................................................126 5.3.3 Implications for Training and Education....................................................................129 5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................129 5.4.1 Toward a More Integrated Theory..............................................................................129 5.4.2 Emotion and Decision-Making...................................................................................130 5.4.3 Emotion and Nested Time Scales ...............................................................................131 5.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................132 5.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133 5.5 Beyond Science..................................................................................................................134 5.5.1 Existentialism..............................................................................................................134 5.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135 5.6 References ..........................................................................................................................137
Section II. Verification and Validation of OBTE as a Service System..................................142
Chapter 6. Initial Impressions of Participation in CATC.............................................................143 6.1 Methods..............................................................................................................................143 6.1.1 Participants..................................................................................................................143 6.1.2 Procedure ....................................................................................................................143 6.1.3 Analyses......................................................................................................................144 6.2 Results ................................................................................................................................144 6.3 Implications for Service System Development: Peer Review ...........................................146 6.4 References ..........................................................................................................................147
Chapter 7. Local Development of Measures of Effectiveness .....................................................149 7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training?.................149 7.2 Measure Development Process ..........................................................................................150 7.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ...................151 7.4 Implications........................................................................................................................156 7.5 Conclusions ........................................................................................................................158 7.6 References ..........................................................................................................................159
Table of Contents xii Asymmetric Warfare Group Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....160 8.1 Methods..............................................................................................................................160 8.1.1 Participants..................................................................................................................160 8.1.2 Procedure ....................................................................................................................160 8.1.3 Analyses......................................................................................................................161 8.2 Results ................................................................................................................................163 8.2.1 Behavior of DS ...........................................................................................................163 8.2.2 Behavior and Performance of Privates .......................................................................165 8.2.3 Patterns of Communication ........................................................................................168 8.2.4 Potential Influence of Instructor Behavior on Performance of Privates.....................170 8.3 Implications for Service System Development..................................................................171 8.3.1 Verification of OBTE .................................................................................................171 8.3.2 Validation of OBTE....................................................................................................172 8.4 References ..........................................................................................................................173
Chapter 9. Impact on Rifle Marksmanship Training....................................................................174 9.1 Behavioral Data Collection During Basic Rifle Marksmanship........................................174 9.1.1 Method........................................................................................................................174 9.1.2 Assessment..................................................................................................................175 9.1.3 Results An Overview...............................................................................................177 9.1.4 Evidence for Influence of OBTE................................................................................178 9.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................180 9.1.6 Behavior of Privates....................................................................................................182 9.1.7 Patterns of Communication ........................................................................................186 9.1.8 Summary.....................................................................................................................186 9.2 Attitudes Toward an OBTE in Basic Training...................................................................187 9.2.1 Method........................................................................................................................187 9.2.2 Results.........................................................................................................................187 9.4 References ..........................................................................................................................191
Chapter 10. Influence of CATC in an Operational Setting ..........................................................192 10.1 Methods............................................................................................................................192 10.1.1 Participants................................................................................................................192 10.1.2 Procedure ..................................................................................................................192 10.1.3 Analyses....................................................................................................................193 10.2 Results ..............................................................................................................................193 10.2.1 Downstream Impact on Marksmanship ....................................................................193 10.2.2 Downstream Impact on Training in the Units ..........................................................194 10.2.3 Downstream Impact on Self Efficacy.......................................................................195 10.3 Implications for Service System Development: Validation.............................................196 10.4 References ........................................................................................................................197
Chapter 11. Implications for Service System Development.........................................................198 11.1 Lessons Learned about Transfer of OBTE.......................................................................198 11.2 Implications for Service System Development................................................................199 11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........199 11.2.2 Further Development of OBTE as a Service System ...............................................199 11.2.3 Further Verification and Validation of OBTE..........................................................201 11.3 References ........................................................................................................................203
xiii Riccio, Diedrich, & Cortes (Eds.) Asymmetric Warfare Group Section III. Further Development of OBTE as a Service System ..........................................206
Chapter 12. Development of General Measures for Students ......................................................207 12.1 Intent ................................................................................................................................207 12.2 Performance Measure Development Process...................................................................207 12.2.1 Phase One: Define Performance Indicators (PI).......................................................207 12.2.2 Phase Two: Translate PI into performance measures...............................................208 12.2.3 Phase Three: Measure refinement.............................................................................208 12.2.4 Phase Four: Retranslation of Measures ....................................................................208 12.3 Product of Measure Development....................................................................................209 12.3.1 Learner Perception of the Instructor and Course ......................................................209 12.3.2 Learner Engagement .................................................................................................211 12.3.3 Student Relationship with Teacher ...........................................................................212 12.3.4 Student Results .........................................................................................................214 12.3.5 Self-Report Measures ...............................................................................................216 12.4 Conclusion........................................................................................................................217 12.5 References ........................................................................................................................217
Chapter 13. Adapting OBTE in a Classroom Environment .........................................................219 13.1 Intent ................................................................................................................................219 13.2 Observing OBTE in the Classroom Environment............................................................219 13.2.1. Participants...............................................................................................................219 13.2.2. Procedure .................................................................................................................220 13.2.3. Measures ..................................................................................................................220 13.3 Utility of OBTE Measures in a Classroom Environment ................................................220 13.3.1 Generality of Measures.............................................................................................220 13.3.2. Implications for Improvement of Measures.............................................................221 13.3.3 Implications for improvement of course design .......................................................222 13.4 Use of 360 Reviews for Collaborative Reflection..........................................................223 13.4.1 The Role of a 360 Review in OBTE.......................................................................223 13.4.2 Narrative of a Participant Observer ..........................................................................225 13.5 Learning, cognitive load and motivation..........................................................................228 13.5.1 The NASA Task Load Index as a subjective measure of student workload.............228 13.5.2 Results.......................................................................................................................229 13.5.3 Implications ..............................................................................................................230 13.6 Conclusions ......................................................................................................................230 13.7 References ........................................................................................................................231
Chapter 14. Organizational Climate and Creation of Durable Change ........................................233 14.1 The Need ..........................................................................................................................233 14.2 Initial Indications of Possible Resistance to Change .......................................................234 14.3 Models and Considerations for Sustainable Change........................................................235 14.3.1 The Change Transition Period ..................................................................................235 14.3.2 Organizational Culture..............................................................................................237 14.3.3 Clarity of Mission and Shared Understanding..........................................................237 14.3.4 Relevant Observations During the Current Investigation.........................................238 14.3.5 Organizational Support and Incentives.....................................................................238 14.4 Conclusions ......................................................................................................................239 14.5 References ........................................................................................................................239 Table of Contents xiv Asymmetric Warfare Group Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................242 15.1 Background ......................................................................................................................242 15.2 An Emerging Consensus ..................................................................................................244 15.2.1 What Part to Balance?...............................................................................................244 15.2.2 Improving Training, by Design ................................................................................245 15.2.3 Increased Use of dL and Dependence on Self-Development ...................................246 15.2.4 Future Orientation, Unknown Requirements............................................................247 15.2.5 The Quality Instructor Challenge .............................................................................247 15.2.6 Purpose and Design are Key.....................................................................................248 15.2.7 A Natural Advantage ................................................................................................249 15.2.8 Task Specialization or Generalized Competency .....................................................249 15.3 Conclusion........................................................................................................................251 15.4 References ........................................................................................................................252
Epilogue. Integration of Leader Development, Education, Training, and Self-Development .....254 Toward Values-Based Standards for Army Doctrinal Requirements ......................................254 Nested Standards and Quality Assurance.................................................................................256 Needs and Opportunities for Staff & Faculty Development ....................................................259 A Role for Science and Measurement .................................................................................259 Toward Best Practices in Instructor Education....................................................................260 Critical Considerations for Further Scientific Investigation ....................................................263 The Necessity of Long-Term Studies ..................................................................................263 False Dichotomy of Objective-Subjective...........................................................................264 Clarity About What Is Evaluated.........................................................................................265 Next Steps ............................................................................................................................266 References ................................................................................................................................268
Section IV. Appendices...............................................................................................................270
Appendix A. OBTE Principles & Practices: Instructor Measures................................................271 A.1 Genesis of Formative Measures for Instructors ................................................................271 A.2 Principles of Outcomes-Based Training & Education ......................................................272 A.3 Guide to Using Measures of Instructor Behavior..............................................................276 A.4 Complete Menu of Instructor Measures............................................................................279
Appendix B. OBTE Principles & Practices: Student Measures ...................................................318 B.1 Guide to Using Measures of Student Behavior .................................................................318 B.2 Complete Menu of Student Measures ...............................................................................319
Appendix C: A Commanders View of Outcomes-Based Training and Education.....................340 Summary ..................................................................................................................................340 Definition.............................................................................................................................340 Description...........................................................................................................................340 Elements of OBTE. ..................................................................................................................341 Developing the Outcomes....................................................................................................341 Developing the Training Plan..............................................................................................341 Conducting Training............................................................................................................342 How Training is Assessed....................................................................................................344 Conclusion................................................................................................................................344 xv Riccio, Diedrich, & Cortes (Eds.) Asymmetric Warfare Group Appendix D: Warrior Ethos..........................................................................................................345 Analysis of the Concept and Initial Development of Applications..........................................345 Current Understanding of Warrior Ethos.............................................................................345 Purpose.................................................................................................................................348 Approach..............................................................................................................................348 Expansion of the Definition of Warrior Ethos.....................................................................348 The Tenets of Warrior Ethos ...............................................................................................349 Clarifying the Definition of Warrior Ethos..........................................................................351 Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353 References ................................................................................................................................355 Supplementary Work Product from Warrior Ethos Project .....................................................355