Sociocultural Factors in The Intelligence Consumer's Reasoning About Analytic Confidence

Sociocultural factors in the intelligence consumers reasoning about analytic confidence
DAVID J. SAAB
College of Information Sciences and Technology, The Pennsylvania State University, University Park, PA 16801, Email: dsaab@ist.psu.edu
WILLIAM MCGILL
College of Information Sciences and Technology, The Pennsylvania State University, University Park, PA 16801, Email: wmcgill@ist.psu.edu
on the broadest issues of the nature and intentions of other countries and the existence and characteristics of broad historical trends, peoples beliefs are determined more by their general worldviews, predispositions, and ideologies than they are by the sort of specific evidence that can be pieced together by intelligence.i Robert Jervis The clash between the intelligence community (IC) and policymakers has gone on seemingly forever, and it seems unlikely that this will change. This clash is multidimensional and has an impact on the levels of confidence policymakers place in the analyses provided by the IC. Robert Jervis explores why this clash happens and concludes that both producers and consumers of intelligence have cognitive predispositions that result from and perpetuate confirmation biases.ii The idea that humans have a confirmation biasthe seeking or interpreting of evidence in ways that are partial to existing beliefs, expectations, or a hypothesis in handis not new.iii It has been described anecdotally since Thucydides (c. 460-395 BCE) wrote his history of the Peloponnesian War. We tend to see what we expect to see. Individuals assimilate and evaluate information through their mental models, which, in this paper, we refer to as schemas.
Saab & McGill
Sociocultural Factors in Reasoning
The work of intelligence analysts is fundamentally cognitive in nature.iv They identify problems, generate and evaluate hypotheses, identify and assess information, recognize patterns, aggregate information, and provide judgments, forecasts and insights to consumers of intelligence. They often conduct their work in groups, making their analyses subject to social processes.v The work of consumers regarding the intelligence produced is also fundamentally cognitive in nature insofar as they have to assess and judge the intelligence and make decisions about policies and actions. Consumers are also subject to social processes, conducting much of their decision making in political contexts, where competing or conflicting decisions need to be negotiated or justified to others. The cognitive and social dimensions of intelligence production and consumption are not completely separable, nor can they be considered independently. They are dynamically intertwined, and each influences the other. Their interactivity constitutes a cultural phenomenon. We develop the mental models related to intelligence in our attempts to understand wicked problems, which have many possible root causes and many possible, and usually unpredictable, outcomes.vi Our mental models regarding intelligence information are never merely disaggregations of data that have been resynthesized.vii Our mental models involve reasoningmeaningful ways of linking patterns of dataand without them we would be overwhelmed by an incomprehensible volume of information. Reasoning is a complex phenomenon that includes not only logic and rationality, but also emotion and motivation. Like our mental models, reasoning is always situated within a field of considerations, i.e., always contextualized and temporalized. Johnson and Berretts thorough examination of including cultural information as part of the analysis of intelligence suggests that lack of cultural data in mental models is a problem not only for analysts but also for the policymakers they support.viii
2
Saab & McGill
The type of cultural data described by Johnson and Berrett is more correctly referred to as cultural profile data. It derives from a classical notion of culture as a system of shared beliefs and values of a cultural group. The construction of a cultural profile is based on a series of dimensionsvalues dimensions in particular. It is a good way to begin an exploration of cultural differences, but it is limited in that it can apply only to cultural groups. One cannot derive the cultural attributes of an individual from such a profile because individuals, in fact, are members of many cultural groups.
SOCIOCULTURAL FACTORS IN REASONING

Dimensional models of culture only provide a starting point for understanding the cultural basis of individual interactions. An emergent model of culture, although complex, provides us with a way of understanding individual behavior within cultural contexts. Applying the insights of an emergent culture model to the reasoning employed by a decision maker regarding an intelligence analysis reveals that there are several sociocultural factors involved in a consumers reasoning and decision-making. These sociocultural factors involved in reasoning are interdependent patterns of objects in the world, individual cognition, and social interactions based in cultural understanding. These patterns are identified in Figure 1 as (1) conferring trust, (2) confirmation bias, (3) social networks, (4) cultural background/identities, (5) the media in which the intelligence is presented, and (6) an assessment of the intelligence producers understanding of the issue. Each of these patterns factor into the reasoning process of the intelligence consumer and shape the levels of confidence they have in the intelligence presented.
Saab & McGill
The purpose in our designating these sociocultural factors as patterns is to convey a sense of their complexity. Each can be considered from different perspectives. For example, trust can be something that is conferred based on a single datum or a series of data. It can be understood as an inherent personality trait or disposition, e.g., a trusting person. It can be conferred because it resonates with ones own confirmation biases or because the person to whom it is given is part of ones social network. It could be conferred because of the timing of the presentation of intelligencebecause the consumer was just in the mood. Like intelligence analysts estimates of likelihood, these sociocultural factors rarely lend themselves to simplistic reduction. They are complex patterns that are linked schematically in our cognition and sociocultural experience.
Figure 1. Sociocultural Factors in Reasoning

4
Saab & McGill
Their schematic nature implies that their influence on reasoning isnt wholly deliberative, that often they function as automatic processors of experience because they are so strongly patterned and embedded within cognition. Their schematic nature allows us to process a lot of information, especially contextual information, that would otherwise overwhelm us or slow us down so much that we wouldnt be able to function in any realistic way. But their schematic nature also leads to undesirable outcomes such as confirmation bias and blind spots.
SCHEMAS
Describing these sociocultural influences as schematic leads directly to the question: So, what exactly are schemas? We want to examine this question to understand better how schemas contribute to the formation and enactment of the sociocultural factors identified in Figure 1. This examination will also lead us to new avenues of exploration related to culture. It will move us away from the classical notion of culture as a system of norms, values and beliefs, to an understanding of culture as an emergent phenomenon that provides both stability and adaptability for both groups and individuals. Schemas are strongly connected networks of cognitive elements, having a bias in activation through repeated exposure to the same or similar stimulus, but they are not rigid and inflexible. Strauss and Quinn, in constructing their cognitive theory of cultural meaning, describe schemas as networks of strongly connected cognitive elements that represent the generic concepts stored in memory.ix DAndrades original exposition on cognitive anthropology a couple years earlier had an expanded set of qualities to describe schemas as flexible configurations, mirroring the regularities of experience, providing automatic completion of missing components, automatically generalizing from the past, but also continually in modification, continually adapting to reflect the current state of affairs.x Schemas facilitate our
5
Saab & McGill
cognitive functioning, including use of our knowledge, in a world overflowing with all kinds of patterns in newly emerging contexts. Schemas function as filters of experience. They help us to cope with potential information overload by enabling us to focus on salient parts of our experience. But because of the strong connections of elements and their clustering, schemas also enable us to fill in information that may be missing from our experience. Schooling, for example, has a number of associated elementsteachers, peers, desks, books, reading, writing, lecturing, and so on. When someone mentions school or schooling, there are a variety of schemas evoked in cognition based upon our lived experience with the same. Perhaps we think of completing our assignments if we are students, and perhaps we think of devising and delivering lessons if we are teachers. Simply mentioning the name of something is often enough to activate schemas associated with it. Schemas help to fill in the ambiguous or missing information because the associated cognitive elements are more likely to be activated by the initial stimuli. People recall schematically embedded patterns more quickly and more accurately than schema-inconsistent patterns or events. In fact, schemas hold such sway in our cognition that people may falsely recall schematically embedded events that did not occur.xi They are more likely to recognize patterns embedded in existing schemas because of repeated activation of the schemas. This repeated activation evokes expectations within cognition and easy recognition of contradictory or challenging information that do not conform to those expectations formed as part of the existing schemas. Patterns that are orthogonal to our schematic expectations are much less likely to be noticed or recalled. It is easy to see how schemas contribute to confirmation bias. Confirmation biases can be perceptualwe tend to perceive what we expect to perceive. When we encounter ambiguous or
6
Saab & McGill
unexpected information, we try to organize it through existing schemas. We will dismiss, discard or ignore elements of that information that dont conform to our expectations. Confirmation biases can be evaluativewe focus on the salient elements within the data pattern that are consistent with existing schemas and use those to justify our analyses and conclusions. The ability to do this can be quite advantageous in some situations. A fighter pilot or a ship commander, for example, needs to make split second assessments of potential threats and decide within moments what course of action they will take. They engage in what Gary Klein & Caroline Zsambok has called naturalistic decision making,xii which stands in contrast to the more deliberative rational decision making. People engaged in naturalistic decision making use abductive reasoning, where rational decision making is the product of deductive reasoning.xiii Since schemas also function as pattern-completion processors, they allow us to generate expectations that we use in conjunction with clues to orient ourselves to the environment. Continued exposure to the same or similar patterns will eventually become meaningful in the sense that it will be associated with other ontologized schemas. We rarely encounter isolated patterns in the world; our experience is more complex than that, and our schemas reflect that. As new patterns appear, new clues for action are generated, which activate other schemas and generate new expectations that enable our active orientation within the world. Experience of varying contexts facilitates the strengthening of schematic conceptual connections and their extension to other concepts. Schemas are cognitive entities. They help us to process information. But they are also cognitive patterns that enable the recognition of entities and patterns in the external world. It would be inaccurate to say that schemas are separable from culture, for that would imply that culture consists solely of the external world structures outside the individual. The discussion of
7
Saab & McGill
schemas marks a shift away from the focus on deliberative and explicit cognitive processes, which mirror the ways we deal with language in cognition, to thinking and cognition as automatic and implicit. It is the shift away from symbolic processing models of cognition towards connectionist and enactivist models of cognition. Cultural schemas Schemas, as complex cognitive associations, are intrapersonal structures. The objects or events that are manifest outside individual cognition, the entities in the external world, are extrapersonal structures. Culture consists of the interplay between the intrapersonal cognitive structures and extrapersonal structures such as systems of signs, infrastructure, environment, social interaction, and so on. The intrapersonal and the extrapersonal are different and distinct, but closely interconnected. They are not isolated from one another. Rather, they separated by a permeable boundary. Culture encompasses both intrapersonal schemas and extrapersonal structures, emerging from the interplay between them. It is through this interplay that we can see that some of the intrapersonal cognitive structures called schemas are shared. The sharing of schemas does not require people to have the same experiences at the exact same time and place, rather that they experience the same general patterns. As humans, we organize our experiences in ways that ensure ease of interaction, coordination of activities, and collaborative interaction. Because we organize our experiences in particular ways, people in the same social environment will indeed experience many of the same typical patterns. In experiencing the same general patterns, people will come to share the same common understandings and exhibit similar emotional and motivational responses and behaviors. However, because we are also individuals, there can be differences in the feelings and motivations evoked by the schemas we hold. The learners emotions and consequent
8
Saab & McGill
motivations can affect how strongly the features of those events become associated in memory.xiv Individuals will engage the external world structures and experience the same general patterns. Similar stimuli and experiences will activate similar schemas. It is in that sense we considered them shared schemas. Its their quality of sharedness that makes them a dimension of the cultural. We share the intrapersonal dimensions of culture when we interact with others. In sharing these intrapersonal dimensions, schemas are activated. Activation evokes meanings, interpretations, thoughts, and feelings. We make meaning of our experience. The cultural meaning of a thing, which is distinct from the personal cognitive meaning, is the typical interpretation evoked through life experience, with the acknowledgement that a different interpretation could be evoked in people with different characteristic life experiences. In some cases our experience is intracultural, where we share a similar cultural frame. In other cases our experience is intercultural, where we are sharing different cultural frames. The meanings evoked by one person in relation to a particular extrapersonal structure may not be the same as those evoked in another. In fact, the meanings evoked may not be the same within the same person at different times, for they may experience schema-altering encounters in the interim. The ways in which we share these intrapersonal dimensions of culture makes each person a junction point for an infinite number of partially overlapping cultures. The notion of schemas has been continually reaffirmed in the IC.xv Citing Heuer, the [Intelligence] Tradecraft Primer strongly asserts the influence of schemas (i.e., "mental models") as contributing to confirmation bias.xvi We might say, for example, that confirmation bias is cultural bias, and that it not only crystallizes an individual's perceptions and understandings but also crystallizes cultural perceptions and understandings. Too strong a confirmation bias can
9
Saab & McGill
result in the distortion of intelligence by the analyst, or it can result in rejection by the policymaker. Neither is necessarily desirable and neither takes place in a vacuum devoid of meaning making. However, we cannot get around the use of schemasthey are a natural and integral part of our cognitive abilities. They help us to recognize familiar patterns and help us to interpret the data patterns we encounter based on our experientially developed knowledge patterns. And, because they already have a bias in activation, more experience of the same patterns will produce stronger schemas, leading to more prevalent confirmation bias. Confirmation bias becomes problematic when its negative pragmatic effects are realizedthere is an attack for which we are unprepared or harmful event that we cannot effectively mitigate. Sense making and Meaning making In Gregory Trevertons Foreword to David T. Moores Sensemaking: A Structure for an Intelligence Revolution, he advocates sense making as an integral part of a new intelligence paradigm that breaks from the Cold War practice of Analysis of Competing Hypotheses (ACH).xvii Because intelligence analysts today are dealing more with activities and transactions rather than things, the old strategy of separating something into its constituent elements should give way to a sense making strategy that will sometimes disaggregate events, sometimes aggregating multiple perspectives, always entertaining new hypotheses.xviii The notion of sense making advocated by Moore is consistent with our description of schemas: a set of philosophical assumptions, substantive propositions, methodological framings, and methods.xix Sense making, as Moore describes it, is an expansive concept. It includes notions of salience, association, process, interpretation and integration of analysis, synthesis into and awareness of an ontological whole. Sense making is a motivated, continuous effort to understand connections (among people, places, and events) in order to anticipate their
10
Saab & McGill
Figure 2. Ontologization: The Phenomenological Character of Information
trajectories and act effectively.xx Saab and Riss explored the interrelationship of these elements without being overly reductive (see Figure 2),xxi as Klein et al. advocated.xxii Accordingly, and consistent with Saab and Riss, we want to suggest a refinement to Moores idea of sense making to distinguish it from meaning making. Sense making and meaning making are entwined processes, but distinguishing the two helps us to better understand the connection between individual schemas and cultural schemas, and also helps to understand the distinction between cultural background/identity and social network in Figure 1. The sense making process associated with data is both objective as well as subjectivebesides its binding to the pattern, as part of its objective facet, it depends on the interpreters knowledge and the situational context in which it is interpreted. To express the difference between the subjective and the objective side we distinguish sense making, as a plainly subjective process, from meaning making as the social dimension of information.
11
Saab & McGill
The sense making of patterns allows for the discernment of data, which provide clues for possible action as an orientation bias for the subject. We cannot say how much sense we must have to discern data within a pattern since this is a continuous process and depends on the amount of time we invest in it. The longer we reflect on a pattern, the more sense we can associate with it, the more clues it provides to us. Our schemas, which are necessarily tied to previous experience, help generate these clues. Though sense making is a process of an individual, it also takes place in a shared environment that produces similar experiences in cognitive agents who are embedded in similar sociocultural contexts. These two sociocognitive factorssharedness and embeddednesslead to a streamlining of the individual sense making processes towards a cultural meaning making process. This streamlining consists in a continuous, mutual adaptation of individual sense and social meaning via communication and collaboration. It is the manifestation of hermeneutic process where the player is acting within the context of the game, but is simultaneously aware of the rules of the game that are shared among all the players. This requires that the individual subject is usually aware to what degree the sense that he or she gives some pattern deviates from the average sense that others might give and which are reflected in others schemas. In pattern completion, schemas function, in some sense, as flexible experience-based filters that make us attend to the salient features of a pattern while filtering out the non-salient. Schemas role in regulating what is salient and non-salient is essential to our understanding of sense making and meaning making. What is salient in a particular context depends on the focus of our sense making and meaning making. This does not mean that that which is not salient at any given moment is not being processed as part of the informational context, simply that it is subsidiary to the salient. It can only be meaningful if we have the appropriate schemas for
12
Saab & McGill
recognizing its salience. Saab and Risss assertion here contrasts with Fred Dretskes assertion that information does not have to be meaningful to a subject to be information; it can simply reflect a stable pattern, such as relevant contextual or background information available to a subject.xxiii However, being relevant means being salient to some degree, which does not imply that what is less salient is not meaningful, only that it is subsidiary to that which has greater salience at that moment. The enactment of schemas reveals that the patterns we perceive are always meaningful in that we are able to identify what is more salient and comes to occupy our focal awareness, and what is less salient and comes to occupy our subsidiary awareness.xxiv Dretske was correct in that we should not confuse information with meaning, but Saab and Riss also assert that we should not confuse meaning with salience, nor should we mistake schematic for not meaningful. In terms of salience and schemas, the question that has relevance to the desires of the IC to be able to identify and connect relevant information from myriad sources to target bad actors, prevent attack or mitigate harm is this: How do we acquire an understanding of what is salient, which become integral to our schemas? As agents in the world, our experiences are organized in ways that ensure ease of interaction, coordination of activities, and collaborative interaction. Certainly, we develop schemas individually, as they are part of our cognitive networks. But we also develop cultural schemas, which are those schemas we co-develop and co-create with others. The notion of what becomes salient often depends on our sociocultural interactions with others, though it may also arise out of purely personal experience as when encounter something that is about to cause us injury. It is this process of creating a mutual cultural understanding of patterns within our experience that we call meaning making. It is sense making on a sociocultural level, where the awareness of others knowledge capacity allows us to refine the sense of a thing
13
Saab & McGill
into meaning.xxv Both sense making and meaning making are interdependent facets of the interpretative process that allow us to ground perceived patterns in data tied to the physical world and the other allowing us to reconfigure the associations of these patterns in ways that can be communicated meaningfully in context. The entwined processes of sense making and meaning making involve a back and forth between knowledge patterns and data patterns within a continuous dance of analysis, expectation, synthesis and communication. Applying Saab and Risss phenomenological model to intelligence issues, we can recognize that while analysts and consumers may perceive the same general patterns of data (extrapersonal structures), their knowledge patterns, mediated through intrapersonal schemas, may have different motivations and their associations may need to be reconfigured to communicate them meaningfully to other consumers, policymakers and decision makers.
REASONING
Once people have started thinking about a problem one way, the same mental circuits or pathways get activated and strengthened each time they think about it. This facilitates the retrieval of information. These same pathways, however, also become the mental ruts that make it difficult to reorganize the information mentally so as to see it from a different perspective.xxvi Johnson and Berrett Reasoning is the capacity human beings have to make sense of things, to establish and verify facts, and to change or justify practices, institutions and beliefs.xxvii There are many types of reasoning.xxviii Reasoning is as much a social process as it is a cognitive one. With respect to our model of sociocultural factors in Figure 1, reasoning involves considering extrapersonal structures such as communicative documents, social networks, and overt displays of trust. Extrapersonal structures are those objective things that we presume to exist independent of ourselves and not dependent upon any particular individual for their existence. Reasoning also
14
Saab & McGill
employs intrapersonal schemas that shape our understanding of those extrapersonal structures. Intrapersonal schemas are cognitive patterns that comprise knowledge. Reasoning, therefore, is an interplay of the extrapersonal and intrapersonal, which is the very definition of cultural. Reasoning emerges from the entwined processes of sense making and meaning making as we move between data patterns and our knowledge patterns. Reasoning applies logic and abstraction to data patterns that constitute the extrapersonal structures of the world and to knowledge patterns that manifest schematically within cognition as intrapersonal schemas and become shared as cultural schemas through sociocultural experience. While data patterns are considered to be "objective," the patterning of the data elements is a cultural phenomenon. Different people can look at the same data elements and derive completely different patterns from it, with attendant associations to other data elements and patterns. The data patterns we sense in the world activate knowledge patterns, which give meaning to those data by way of logic and abstraction, which are in turn dependent upon the shared cultural schemas that develop as part of our sociocultural experience. However, we never arrive into a context as blank slates. Just as data patterns activate knowledge patterns, so too do knowledge patterns orient us toward particular data patterns. Inasmuch as we are immersed in a world of data patterns, we are also projecting our knowledge patterns onto the world by way of schemas. The reasoning in which we are able to engage is regulated by schemas. If we consider a geocentric view of the universe, popular for 1500 years until the early modern agewhen Copernicus proposed a mathematical model of heliocentrism in the 16th Century, Kepler expanded upon the model, and Galileo made confirmatory observations a century laterwe can see that the possible paths of reasoning for scholars, in general, were restricted by the schemas they held. Our experience informed us that we werent moving, but rather the sun and night stars
15
Saab & McGill
were. Since the Earth seemed stationary, and we saw the sky and stars move, it made basic intuitive sense that the Earth was stationary and everything else orbited around it. These schemas were not only maintained across generations, but were codified into the sacred texts of Western civilization. To question the logic of the geocentric view was an affront to the social network of scholars who studied celestial mechanics. It was also an affront to the authority of the sacred text and, by extension, God. To question the logic of geocentrism was to question the logic of the Churchs authority. So, we had both experienced-based cognitive schemas and the authority of social networks constraining the logic of possibilities regarding a non-geocentric universe. As Thomas Kuhn, in The Structure of Scientific Revolutions, observed, the number of anomalies continued to build within a geocentric theory of the universe.xxix Planets moved in strange waysseemingly reversing their orbit around Earth. Still, people hung on to their schemas and beliefs. Even in the face of new evidence and a simpler, more elegant explanation of heliocentrism. The shift to heliocentrism as the dominant theory in celestial mechanics took centuries. In order for this to happen, people needed to be open to recognizing new data patterns, willing to adapt their existing knowledge patterns and schemas, develop new schemas, and create a consensus about the patterns they were seeing in the world (or, in this case the night sky). Once they were willing and able to recognize these new data patterns and incorporate them into their knowledge patterns, the abstractions they created were no longer constrained by the logic they had been using. The new theory relied on a different set of abstractions and new logical connections that made them explicable and meaningful. When it comes to the IC, the connection between reasoning and culture can be understood in terms of a cultural topography advocated by Johnson and Berrett.xxx Each of us is inescapably immersed in the cultural. We are making any number of schematic assumptions at
16
Saab & McGill
any given moment that are subsumed as the contextual part of our experience. We can say that these assumptions contribute to an implicit reasoning in which we engage as part of our experience. Cultural groups make different assumptions about context and focus on different salient elements of social interactions and environments. Explicit reasoning takes the form of explanation, e.g., answering questions, that integrates implicit cultural assumptions about context that might not be articulated. Implicit and Explicit Reasoning Implicit and explicit reasoning have been formally theorized as abductive, inductive and deductive reasoning. Intelligence analysis [is] a process of ceaseless discovery in a nonstationary world, process involving evidence in search of hypotheses (through abductive reasoning), hypotheses in search of evidence (through deductive reasoning), and evidential tests of hypotheses (through inductive reasoning), all going on at the same time.xxxi Abductive reasoning is schematic and occurs without conscious deliberation. Abductive reasoning is what we employ most of the timewe are prompted by a few key pieces of information and leap to a decision, conclusion or understanding of the situation. Abductive reasoning is developed through experience, where the strength of the schematic connections becomes stronger over time. Abductive reasoning usually remains unarticulated, and will take the form of inductive reasoning when made explicit. Explicit reasoning in the form of deduction or induction or both can be formalized. People can be trained to reason in particular ways about particular things. We do this all the time in educational and training settings. Trained forms of reasoning often define cultural boundaries, as the training involves ontological commitments to particular entities and phenomena in the world. The training of an academic, for example, is different from the training of a soldier with
17
Saab & McGill
respect to their reasoning about the world. As individuals they might overlap in terms of their cultural identities in other waysas Bostonians, as athletes, as parents, as photographers, and so on. Put them in a situation where they are cultural outsiders, their training will lead them to interpret the behavior of others in different ways. One might see the raised voices and exaggerated hand gestures of two people in a public marketplace as the expected interaction for commercial exchange (i.e., haggling), where the other might see it as a potentially threatening activity that requires defensive preparation (i.e., a distraction preceding a planned attack). The former might let the phenomenon slip out of his awareness as something unremarkable, whereas it might prompt a more vigilant awareness of the entire area in the latter. The method of Alternative Competing Hypotheses (ACH) in the IC is an example of this type of trained reasoning. ACH requires an analyst to explicitly identify all the reasonable alternatives and have them compete against each other for the analyst's favor, rather than evaluating their plausibility one at a time. The key factor here is reasonable. What is reasonable? What is within the realm of possibility? What considerations are relevant? What have the analysts been trained to see as part of their sense making? And in what ways have they been trained to interpret it as part of their meaning making? Transforming reasoning into a formalized methodology has advantages and disadvantages. The formalization enables faster and more consistent processing of information. However, its codification into method constrains its adaptability and its ability to incorporate new types of information. While the intent of ACH is to overcome these constraints and foster a larger perspective, it is employed by intelligence analysts who are immersed, as Johnson and Berrett accurately describe, in disciplines that emphasize power and wealth as the primary human motivators, leaving underexplored other motivators such as identity, preservation of
18
Saab & McGill
social institutions, alternative value structures, powerful narratives, or perceptions of the security environment distinctive to a persons or groups region and history.xxxii Cultural Reasoning in IC Although people are, by nature, variable and unpredictable, they still need to work with others in social and cultural groups. These groupsand their associated beliefs and structuresare organized according to logical, understandable principles that every person living in the culture must understand, at least intuitively, in order to get along with each other. With some basic study, [others] can also recognize and understand these principles and apply that understanding to their operations.xxxiii Salmoni and Holmes-Eber The producers and consumers of intelligence often have different motivations, different social institutions and networks, different narratives, and perhaps different value structures. All of these factors will contribute to differences in their reasoning about the intelligence they deal with. Analysts use particular intrapersonal schemas to create confidence judgments about their analyses. Those schemas may or may not be shared by the consumers of the analyses. If there is enough overlap of the schemas, the analyst and consumer can be said to share a cultural understanding. This doesn't mean that they will agree, only that each has the capacity to understand the perspective of the other, to understand how the other reasons. If their schemas don't overlap, they may or may not agree about the analysis, but neither one be able to understand the reasoning of the other. Without this understanding of the reasoning involved, their confidence levels are unlikely to align, and if they do it will be purely coincidental. In Figure 1, the factors delineated are interdependent to varying degrees. There is a pseudo-symmetry between social factors (presentation of information & analysis, social networks, trust) and cultural factors (assessment of analysts understanding, cultural background, confirmation bias). It must be kept in mind that this isnt a perfect symmetry, but rather an easy
19
Saab & McGill
heuristic to distinguish between factors that are purely social and those that are cultural, arising from the interplay between the intrapersonal (cognitive) and the extrapersonal (objective).
IMPACT OF SOCIOCULTURAL FACTORS

In this section we examine each of the sociocultural factors identified in Figure 1 in more detail. We proceed by examining each social factor followed by its associated cultural factor. We begin with the inputthe presentation of information and analysisto the consumer, which prompts the assessment of the intelligence analysis and confidence levels of the analysis. The assessment is carried out in a social contextthe social network of the consumerthat is tempered by his cultural background. The reasoning employed by the consumer leads to a conferring of trust, which may be the result of confirmation bias, but which also reveals the possibility of other reasons to confer or not confer trust. The interplay of all of these sociocultural factors is then argued to have an impact on confidence levels of the consumer in the intelligence presented.
PRESENTATION OF INFORMATION AND ANALYSIS

Communicating complex judgments and degrees of confidence in those judgments is best done through conversation among the parties, which demands different mechanisms than simple dissemination of facts. If the mechanisms for interaction are designed only to support the provision of individual pieces of evidence rather than to engage both parties in an extended conversation in which ambiguity and subtlety can be communicated, it is unlikely that either party will be satisfied with these interactions.xxxiv Jeffrey Cooper The presentation of intelligence is an extrapersonal structure that begins the cultural interaction with the consumers intrapersonal schemas. The consumer is not only interacting with the physical report and its content, but also with the intrapersonal schemas of the analysts that were used in producing the report. Reports of intelligence analysis include assessments of the reliability and quality of the source of the intelligence. How source information and evaluation is
20
Saab & McGill
portrayed varies widely across the IC. For example, some agencies may not reveal their sources under any circumstances. They will only report on the closeness of the source to the target (level of activity) and the reliability of the sources information in the past (quality of activity). Intelligence analyses commonly include verbal indicators as part of their uncertainty estimates (e.g., very likely, likely, not at all likely). The imprecision of these terms provides an illusion of communication because people generally assume that everyone interprets the terms consistently and similarly.xxxv Consumers of intelligence are similarly predisposed to ignore the potential variance in interpretation such that expressing uncertainty in verbal form is not likely to be an effective method of communicating between analyst and consumer.xxxvi Dieckmann suggests including some type of numerical reference scale with the verbal indicator. By including a numerical reference scale, the variance in interpretation of the verbal indicators is decreased, as the numerical scale provides greater specificity for more precise meaning making by the consumer. Attaching the verbal indicators to a numerical scale facilitates the evaluability of that numerical information, which have been shown to improve judgment and choice.xxxvii Weiss recommends that presentations of intelligence for assessment highlight key assumptions that reflect concerns driving the analyst and key indicators that underpin the preferred conclusions.xxxviii Broomell and Budescu capture the tension succinctly when they say, While high reliability is crucial, the similarity of the weighting schemes play a much smaller role than the identification of the key cues and their correlations.xxxix Bring to the foreground that which is most salient to the analyst so the consumer can understand the schemas used in developing the analysis. However, Weiss notes that no matter what technique is used, effective communication is constrained by the willingness of the recipient to hear what is being said.
21
Saab & McGill
Whether the presentation includes verbal indicators, numerical indicators, a combination of these, its purpose is to communicate. The consumer is looking for the analyst to ask the same questions he would ask. In other words, the consumer looks to the presentation to see how aligned the schemas of the analyst are with his own. The closer the analysts questions, key assumptions and key criteria are to his own, the more likely it is that the decision maker will believe that the analyst shares his understanding of what is important and salient. It is more likely to create confidence that the analyst shares the same or similar overall goals.
ASSESSING THE ANALYSTS UNDERSTANDING

A natural part of our interactions with others, especially through discourse, is assessing what it is they understand. If we didnt assess what others understand we would likely express ourselves in incomprehensible ways. I know that you know that I know is a reflective process that we all undertake in communicating with others. It is within the discursive context that we share our sense making of entities and phenomena as part of the meaning making process.xl The use of ACH can enhance the professionalism of the analyst in the eyes of the consumer.xli It demonstrates to the consumer that the analyst performed his or her due diligence by engaging in an exhaustive scoping and analysis of alternative scenarios and/or alternative weighting of elements within the analysis to arrive at the judgment contained in the analysis.xlii Using a standard epistemology such as ACH also allows the consumer to draw upon familiar schemas used by the analyst in composing a report. Analyses that include statistical information are generally viewed as having a valid epistemology.xliii There is a presumption that statistical analyses are more scientific and reflective of the real world, i.e., there is a realist presumption that is shared by the analyst and
22
Saab & McGill
consumer. A mathematical analysis is presumed to be objective and less subject to individuals biases. If the consumer is numerically literate, he can assess the analysts understanding of the issue or if they are fudging their analysis. He might accept or reject the analysis, i.e., confer greater or lesser confidence in it. If the consumer is numerically illiterate he might confer trust and thereby greater confidence in the analysis, as suggested by Dieckmanns research. In comparing the confidence of producers and consumers of risk analyses, Budescu, Broomell, & Por were surprised that the consumers nearly always rated the confidence in the analysts judgments lower than the analysts/producers rated it themselves.xliv This lower rating involved translating verbal indicators into numerical ones, and the latter are considered to be more precise. Ambiguity of meaning in verbal indicators (e.g., very likely, large, abrupt) surfaces because intracultural schema variation is normal. Variation in interpretation becomes more pronounced in intercultural contexts where schema variation might be much greater or even incommensurable.xlv Verbal indicators alone are imprecise, which is considered to be a weakness in communicating uncertainty and inductive of illusory communication.xlvi The imprecision is problematic because it engenders variance in interpretation that goes unrecognizedpeople assume that the way they interpret the phrase is the same way others will interpret the phrase,xlvii i.e., employ the same sets of schemas. Researchers of communities where communicating uncertainty is common (e.g., IC, IPCC) often try (or recommend) to quantify uncertaintyto match verbal indicators (likely, very likely, as likely as not, etc.) with a range of numerical probabilities, preferably in percentages.xlviii Researchers assume that this dual approach of providing a numerical probability match to verbal indicators will (1) improve the quality of communication because it provides more information, (2) facilitates agreement in the
23
Saab & McGill
interpretation of phrases, and (3) caters to heterogeneous audiences with varying expertise. Lets examine each of these in light of our understanding of schemas. Researchers are not wrong in that it provides more informationthat is prima facia true. It does not follow, however, that communication will automatically improve. Rather it provides for the possibility of improved communication by enabling greater precision for meaning making. More information will often allow correspondents to express their sense of something with higher granularity and more nuance via greater numbers of connections between data and knowledge patterns. In order for communication to improve, however, the correspondents need more than just information. They need a meta level understanding of how each other are connecting the data and knowledge patterns. They need to have or to already share a set of cultural schemas that allow them to adapt their individual sense making to a sociocultural process of meaning making. They also need the ability to negotiate an agreement around what those patterns mean and how they are connected to a context.xlix Facilitating agreement in the interpretation of phrases can be an outcome of such dual approaches, if agreement about the meaning of the verbal-numerical matching is negotiated as part of the hermeneutic discourse and results in the development of a cultural schema that can be used by producers and consumers in the future. By adopting a common understanding and a standard way of expressing that meaning (i.e., through verbal or numerical indicators, or both), communicative efficacy is improved. The claim that it caters to heterogeneous audiences, however, is contradictory based on our understanding of schemas. Heterogeneous audiences use different schemas to make meaning of the same data patterns, or their schemas might lead them to perceive different data patterns altogether. If they werent heterogeneous, they would share the same cultural schemas enabling
24
Saab & McGill
them to perceive the same data patterns that have the same meaning. In order to improve communication, the heterogeneous audiences would need to revert to a hermeneutic discourse to develop a more precisely aligned meaning for the verbal-numerical matching. Consumers attempt to make meaning of the uncertainty embedded in intelligence reports by analysts. Estimates of uncertainty are attempts to convey the effect of imperfect information on the intelligence analysis. The amount, type, reliability and unanimity of information are factors for consideration in the reasoning process that gives rise to confidence levels about the likelihood of an event. These factors comprise a second-order uncertainty or ambiguityl and are not likely to be explicitly communicated to the consumers by the analysts. This tendency by the analysts comports with the notion that people tend to prefer less ambiguity in probability assessments, and they tend to view more ambiguous assessments more pessimistically and avoid making decisions based on such information.li This doesnt mean that decision makers are adverse to all ambiguity, just too much of it. Providing a sense of how much ambiguity is present in the analysis through the use of numerical scales is useful for the intelligence consumer.lii Numerical ranges are proxy indicators for how much individual sense making and sociocultural meaning making should engage. For example, a large numerical range suggests that the consumer should expand the possibilities of how events may unfold and entertain alternative scenarios and courses of action. A smaller numerical range would indicate the opposite, that the consumer constrain the possibilities of how events may unfold.
SOCIAL NETWORK
Experience will have taught them to place faith in their own judgments. But they will still seek sources of reassurance.liii Robert Jervis
25
Saab & McGill
Contemporary discussion about social networks focuses on the information and communications technologies that have facilitated their maintenance and expansion. But social networks are and have been an integral part of human experience from the very beginning. We are social primates, after all. There is a difference between the social and the cultural, though the demarcation between the two conceptual groupings is more blurry than precise. Generally speaking, the social is oriented towards the external, towards others with whom we share our experiences and understanding. It can be observed by others, and thereby objectified, through our actions and discourse. The cultural is inclusive of both the external and internal, both the social and the cognitive, and consists of the interplay between them. The cultural includes subjectivity, which is impossible to observe yet is necessary for meaning making. The advent and proliferation of information and communications technologies reveals that the social is enacted as part of a networka social network. Our social networks consist of other individuals with whom we engage in collective action, including discourse. It is the discourse that concerns us here. Through discourse, we share the patterns that we recognize in the environment with others who are part of our social networks. This discursive practice within our social networks helps to structure our schemas. Recall the discussion above concerning cultural schemas, sense making and meaning making. Our social networks are the canvas upon which we develop our cultural schemas and the stage where we act them out. Our social network provides the field upon which we can validate our individual sensory experience. This validation results from others also being able to see the same patterns and attribute a similar meaning to them.
26
Saab & McGill
The discourses within a social network are complex expressions of cultural schemas. The discourses reflect cultural boundaries enacted as a social network. The different social networks to which we belong contribute to our identities, which in turn maintain the cultural boundaries of the networks. These boundaries help us determine who is included and who is excluded as part of our social networks, and they have implications for trust, as we will discuss in a later section. It is this phenomenon of maintenance of cultural boundaries and identities that contribute to groupthink. For within our social networks, it is comforting for us to listen to those who share [the same] general values and outlook.liv The discourses of other social networks are based in different beliefs and theories, different values and outlooks. These differences cause us to be skeptical of others discourses, because we know there might be differences in the fundamental assumptions that our respective social networks make. Our social networks might maintain different goals, different motivations, or different tolerances for dissent. When it comes to reasoning about analytic judgments, senior intelligence officials are convinced that their executive branch clients want to read summary judgments, not alternative possibilitiesin part because many of them are uncomfortable with probabilities, and in part because they fear that the less likely alternatives will be leaked to the press and fuel opposition to the policies that are eventually adopted. There is even said to be a culture that discourages changing a conclusion even in the face of contradictory evidence, for fear that such change might confuse the client. Beyond this, the different agencies of the Intelligence community often prefer to paper over differences of opinion, a task that is made more difficult if they are required to agree on a quantitative expression of just how probable they think a statement is to be true.lv Charles Weiss Social networks are the outward manifestation of cultural groupings. The ways in which we behave are understood culturally and enacted socially. Part of socialization into a particular network or group is acculturation. We need to learn appropriate ways of expressing our salient
27
Saab & McGill
thoughts, feelings and motivations to those within our social network and to those outside of it (assuming we are allowed to express them to outsiders at all). The boundaries of inclusion and exclusion for a social network may not be simply binaryin or outbut may consist of overlapping concentric spheres of influencehow close you are relative to context. Analysts and consumers may collectively belong to a large social network called the IC Community. They may belong to social networks that are bounded by organizational or sub-organizational boundaries, such as the CIA, ONI, NGA, Congress, Congressional Committees and Subcommittees. They might belong to social networks that traverse organizational boundaries, like a bowling league, a gamer community or a golf club. We tend to belong to multiple social networks, and each influences our interactions with others in ways appropriate to particular contexts. Our social networks influence the schemas we develop and how we use them; they are the social space where cultural meaning is developed and evoked for our individual sense making. The integration of our experiences from myriad social networks finds meaningful integration as part of our cultural background.
CULTURAL BACKGROUND
Once people have started thinking about a problem one way, the same mental circuits or pathways get activated and strengthened each time they think about it. This facilitates the retrieval of information. These same pathways, however, also become the mental ruts that make it difficult to reorganize the information mentally so as to see it from a different perspective.lvi Richards Heuer Cultural background is a significant shaper of our worldview. We rarely consider culture as influential on our own analyses and decision making because culture is mostly invisible to us. It is just how the world is. We dont notice culture because it is always there. Our biases arise from and are incorporated into culture. Johnson and Berrett articulate three sources of bias in the IC arising from culture: ethnocentrism, expertise, and institutional training.lvii Each of these
28
Saab & McGill
causes us to adopt a particular worldview, to preference particular elements of a context, and to perpetuate a cultural bias. The lack of cultural data in mental models is a problem not only for analysts but also for the policymakers they support.lviii They suggest incorporating four informational facets to intelligence assessment: identity, values, norms and perceptual lenses. While there is value in Johnson and Berretts cultural mapping, it is not what concerns us here. We want to focus on how cultural background influences reasoning about intelligence and how it contributes to confidence levels. None of us are blank slates. From our discussion of schemas and social networks it should be clear that each of us is immersed in multiple cultural contexts and carry with us multiple cultural identities. This array of contexts and identities constitute our cultural background. At once we can see the complexity of cultural background where complex networks of schemas, comprised of complex networks of patterned cognitive elements that reflect a sociocultural understanding as one interacts with the patterns of extrapersonal structures in the world, and which are accumulated, integrated and adapted through ongoing lived experience within environments that include multiple social networks. Cultural background is distinct from social networks in that social networks are determined by whom we do interact with, while cultural background shapes our ability to interact meaningfully with others. Given this complex definition, cultural background in relation to the IC can mean identity, expert judgment, or ways of thinking. Identity connects the individual to the social network by providing a boundary of inclusion for membership. Individuals need to exhibit an understanding and facility with the cultural schemas employed within the social network. Expert judgment draws on the extended experience of the individual and his development of sophisticated schema patterns that he can bring to bear for a narrowed topic of interest. Ways of
29
Saab & McGill
thinking refers to the schematizing of sense making and meaning making processes that connect knowledge patterns and data patterns. Identity is both a cognitive and a social phenomenon. When we identify with a particular group, we are asserting a shared set of cultural schemas. An identity is an ontologization of this set of schemas, a way of quickly communicating the perspective from which we understand the world. This ontologization creates a membership boundary, determining who is included and who is excluded from the group. Expert judgment arises from a depth of experience we often refer to as specialization. This depth of experience enables the formation of complex schemas about entities and phenomena. The more time I spend focusing on particular entities and phenomena, and the more contexts in which I encounter them, the more complex my understanding becomes of them. My cognitive reach allows me to contextualize the object of my focus in more ways than nonspecialists are able. Ways of thinking concerns not only how we think, but also what we think. It is a blending of the sense making and meaning making in which we engage to connect our knowledge patterns with data patterns. Schemas help us navigate the world and help us to discern what is salient among a complex set of data patterns. Our ways of thinking, then, are constrained by the schemas and salience connected to our knowledge patterns as part of the sense making process. The meaning making process shapes the ways in which we connect complex sets of data and knowledge patterns into a communicable form (i.e., information) that others can understand. This sociocultural shaping is the how we think dimension of ways of thinking.
30
Saab & McGill
Policymakers who receive analyses from the IC are sometimes political appointees who come from professions unrelated to intelligence and who have little time to learn about intelligence once in office.lix Because of their lack of understanding regarding the processes by which intelligence assessments are produced and their being unable to grasp the sometimes subtle complexity of the information provided to them, they make decisions that can put them at odds with careerists within the IC. Sometimes political leaders have to oversell their policies, which produces pressures on and distortions of intelligence, and which in turn results in intelligence officials seeing political leaders as unscrupulous and careless, if not intellectually deficient, and that leaders see their intelligence services as timid, unreliable, andoftenout to get them.lx This conflict is often the result of differences in cultural backgroundidentity formed within distinct social and institutional hierarchies, expertise reflecting the different depths of understanding, and different ways of thinking that result from schemas formed in different contexts with differences in relative salience and different connections made regarding the focus of the intelligence. Careerists and politicos arrive at a common juncture from sometimes very different directions. They arrive having developed different cultural identities along the way. The networks of schemas they employ based on that identity may differ significantly, e.g., the politico may be motivated in part by advancing a political agenda that favors his political party, in which case his starting point for analysis is the desired outcome; the analysis only serves as a means of justification for a prior established political goal. The consumer of intelligence may not understand exactly what they want or what intelligence analysts are able to provide. They dont necessarily know how to ask the right questions. The reason for this is that they have a different cultural background than the analysts.
31
Saab & McGill
Within the consumers cultural context, it might be perfectly acceptable to say, Get me everything you have on [Country X], to staffers who are immersed in the same cultural context and can fill in gaps with prior experience concerning intent, motivation and priorities. They may implicitly understand if the consumer wants information that supports a particular position, provides counterarguments to positions of the person with whom theyve just met, or is focused on a particular issue (e.g., natural resources or intrapolitical tensions). Within their own cultural sphere, consumers may not be used to asking the right questions. This becomes problematic when the same vague questions or requests are made interculturally. In intercultural situations, where the consumers and the analysts share fewer schemas, vague questions and requests are a source of frustration for the analyst who needs to know how to narrow the parameters of the intelligence reporting to provide an appropriate and well-formed answer. When engaging with a consumer who is unfamiliar with intelligence analysis, and therefore doesnt know how to ask the right questions, or even what is possible to ask, the analyst needs to probe for the right questions that enable analysis. Without that probing, the analyst may provide intelligence that does not meet the unarticulated requirements and result in the consumers rejection of it. The notion of cultural background raises the issue of adaptability. The reality of threats appearing suddenly from unexpected quarters has gained salience in recent years. It has led some researchers to advocate for greater flexibility of the IC, that the IC should be able to reconfigure itself to changing needs, that the IC should be able to take advantage of various expertise, that the IC should be able to utilize information from myriad sources, and that the IC should enable and maintain a more densely connected communication network among the various social networks and cultural groupings within it.lxi The formation of ad hoc task forces and intelligence
32
Saab & McGill
centers is an attempt to create institutional mechanisms for adaptability. By bringing together analysts with different cultural backgrounds to combat weapons proliferation, engage in counterterrorism, address narcotrafficking and other special topics of interest, the IC enables a broader sense making ability of the data patterns and more complex meaning making among analysts. It allows analysts to create new and hybrid cultural backgrounds and social networks that enhance their ability to recognize and understand potential patterns that had been previously constrained from view by their cultural backgrounds. It facilitates the extension of identity and wider boundaries for group inclusion, facilitates a deepening of knowledge and expertise, and it facilitates different ways of thinking, thereby facilitating greater adaptability. The success of ad hoc teams and task forces (flattened hierarchies) has lead to speculation that having a master dataset of all intelligence information, to which a very wide range of intelligence officials would have automatic access, would be a worthwhile and desirable project.lxii In practice, however, the multiplicity of cultural backgrounds among IC analysts and consumers would make this endeavor quite difficult and unlikely to succeed. Part of the reason for this is the nature of computational artifacts, which require data to be organized according to a single ontologya specification that articulates the types of data and their interrelationships.lxiii Humans dont maintain single ontologies, but rather multiple and sometimes contradictory ones that provide us the ability to use multiple perspectives based on a multiplicity of cultural identities.lxiv Unlike our computational systems, humans are attuned to context and shift effortlessly between multiple ontologies bounded roughly by cultural identities and shared as cultural schemas. In hierarchical organizations, blind spots result from compartmentalization and restrictions on the dissemination of information. In flattened organizations blind spots result
33
Saab & McGill
from unfused horizons (pace Gadamer) and the failure to recognize salient information because of pre-existing schematic structuring of data. In other words, the schematic structuring of data that arises through organizational hierarchies (i.e., cultures) is not eliminated by flattening the hierarchy, because individuals will always organize data in schematic ways flattened organizational structures simply have more schemas from which to operate and assess. The issue is one of salience. Analyses are created based on available data. Some data may be available (e.g., through a centralized dataset), but unless it is also salient as a corroborating or competitive element its unlikely to be useful. There is no easy way around the computational problem. Using a single ontology privileges one set of cultural schemas over another, which inhibits adaptability and rigidifies mental models. Using multiple ontologies enhances adaptability but produces vague mental models. This is not a Goldilocks situation in which the just right level of adapting and mental modeling can be determined; the contextual possibilities are too vast. Rather than trying to create a master data set, the notion of cultural background suggests we consider reframing intelligence sharing using an information ecology metaphor. The nature of the ecology metaphor suggests diversityof structure, of function, of specializationand the ability to reconceptualize the informational ecosystem from different perspectives. Opening things up to a multiplicity of perspectives and inducing pluralism is the opposite of compartmentalization. The blind spots generated by pluralism, however, might be similar to those generated by compartmentalization in that taking a singular perspective will blind one to the salient elements of other perspectives. There needs to be an understanding of the overall ecology and a continual realignment with more specific perspectives, which metaphorically implies a continual adaptation between the
34
Saab & McGill
ecosystem and the particular species within it. This continual adaptation between ecosystem and species is analogous to the adaptability between cultural group and individual. An ecology can be described in many waysbiologically, hydrologically, geologically, geographically, and so on. The description that emerges is dependent upon what elements are salient to the narrative and how those salient elements fit together and fit within the purpose of the narrative. This is what the intelligence analyst doescraft a descriptive narrative that highlights particular elements as salient, creates or establishes their interconnections in a purposeful waywith new information they have gathered (or recall) in light of the new questions that have been posed. The analyst doesnt necessarily have access to the full set of relevant information, and therefore must rely on their cultural schemas to fill in the gaps. A "reputation and trust" architecture might partially resolve the computational problem. For communities to share such information about its members, one of the preconditions set by the architects of the system is that "the requesting community must decide upon the level of confidence it has in each of the responding communities. This level is decided according to the similarity of the communities categorization, the conversion uncertainty imposed by different value domains, or simply by an explicit assertion. The higher the confidence level of the responding community, the bigger the influence of that community's data in the CCR computation."lxv Intelligence centers are meant to not only bring together culturally diverse analysts, but also to extend the boundaries of identity to intelligence consumers, connecting them directly to intelligence producers.lxvi The new boundaries of the social network allow the members of the social network to leverage reputation information (although reputation information can also be leveraged across communities) to establish trust among members of the group.
35
Saab & McGill
CONFIRMATION BIAS
"Policymakers [often wonder] how 'those clowns at Langley' could misunderstand so much of the world...."lxvii Robert Jervis The clash between the intelligence community (IC) and policymakers has gone on seemingly forever, and it seems unlikely that this will change. This clash is multidimensional and has an impact on the levels of confidence policymakers place in the analyses provided by the IC. Jervis explores why this clash happens and concludes that both producers and consumers of intelligence have cognitive predispositions that result from and perpetuate confirmation biases.lxviii Schemas strongly influence what information we are likely to perceive, accept and remember. Information that conflicts with our schemas is more likely to be dismissed, discounted or ignored completely. Simply put, schemas tend to induce a confirmation bias. That schemas facilitate confirmation bias is neither a new idea nor surprising. Schemas develop largely because there is a regularity to our experience; we are repeatedly exposed to the same general patterns throughout our lives. After a certain point we don't need to relearn what a doorknob is or what we can do with it every time we encounter one. Nor are our experiences of things like doorknobs encountered in isolation, they are almost always associated with doors, rooms that are separated by doors, particular activities that occur in the different rooms, motivations and goals for those particular activities, other people's social roles in those activities, their likely motivations and goals for engaging in them, in those rooms, that have those doors, that are opened with doorknobs. The possible sets of associations we could make among the entities and phenomena we encounter is theoretically infinite. It is simply too much information for us to handle in our ongoing experience. Developing schemas is a natural response to our individual need for processing information about others, our environments, and ourselves.
36
Saab & McGill
People are generally quite poor at matching accuracy and confidence calibration in decision making and tend to be overconfident in their own judgments. Overconfidence in a decision can be decreased through increased information processing in social interactions.lxix When engaged in sense making, an individual determines the set of possible data patterns that are relevant to the context. The set of possibilities is both constrained and open: constrained because the individual employs already developed schemas that are focused on particular clues and salient elements of the pattern, and open because there is a flexibility that allows for the reconfiguration of the possibilities given additional information or different contexts. People tend to be overconfident because the schemas they employ tend to confirm what they already know, which they take to be an affirmation of the reality of the extrapersonal structures rather than solely a projection of their own schemas. When people are engaged in meaning making as a part of social interaction, the flexibility of their interpretations becomes apparent as they try to negotiate agreement of what patterns they collectively see and what elements of the patterns are salient. This flexibility casts their earlier sense making into a larger set of schemas brought into the negotiation by others, and calls into question their certainty and confidence in their individual judgment. When the evidence that others develop independently fits with our previously developed schemas, we dont necessarily feel the need to explore that evidence further. Rather, we tend to accept it as confirmation of our expectations and understanding. In other words, we enter the context (e.g., of evidence provision) with already formed expectations based on our individual sense making. If there is enough of an overlap between another individuals (or groups) sense making and our own, we consider it to be a confirmation of what we already knew. Minor
37
Saab & McGill
deviations in that overlap are mostly ignored or discounted as not salient enough, which reinforces the confirmation bias. Schemas, because they are clustered, consist not only of rational or logical modes of thinking, but also include emotional and motivational biases. If a consumer is strongly motivated and/or there is a strong emotional component to their schemas, confirmation bias may lead to the outright rejection of evidence provided by others. This may be retroactively rationalized as the others not having the right information or not being able to see the larger picture. It may be based in motivating ethical frameworks that conflict with one anotherit is the right thing to do regardless of the outcome vs. it is the right outcome regardless of what we have to do. Whatever the source of the schematic dissonance, we are already disposed to accept that with which we agree. Confirmation bias may lead to the rejection of others assessments simply because it doesnt comport with what one already knows and understands to be the case. In social situations, a majority desire for confirmation can inhibit the expression of dissenting viewpoints. This, too, will lead to a type of confirmation bias known as groupthink.
TRUST
"...the extent to which advice is followed depends on when that advice is received, and how confidently it is expressed."lxx Caroline Wesson Trust is neither completely synonymous nor completely separable from confidence in analytic judgments. Trust is often associated with confidence, but it can be conferred because of presumed authority or expertise. It can derive from long-term interpersonal interaction or from contextual factors. It can be extended to intranetwork/intracultural group members with whom one has no interpersonal relationship or to extranetwork/intercultural group members who we believe share similar motivations and goals. Trust is not a static state of mindwhether and how
38
Saab & McGill
much trust is conferred can change. It can be conferred presumptively to those who have caused me no harm and to those who have contributed to my success or well-being. Trust is both rational and emotional, which can make it difficult to measure, but which reveals its essential integration with schemas. We are more likely to trust someone who agrees with us and shares our cultural assumptions and understanding of the world. This isnt to say that sharing cultural assumptions causes trust, for one can share a great many cultural schemas with someone and still not trust them, e.g., because of an interpersonal or intranetwork rivalry. Rather, sharing cultural schemas lowers potential barriers to conferring trust. The sharing of cultural schemas means that we are likely to share a similar understanding of the entities or phenomena under investigation as well as a similar reasoning as to how they interact and fit together. We will often arrive at the same conclusions. The similarity of ontological commitments, of reasoning and of conclusions reinforces our confirmation bias. We confer trust in these circumstances based on a confirmation bias, since we tend to trust our own judgments implicitly in most cases. Trust can be conferred regardless of schema overlap, but it is more readily conferred with greater overlap. The barrier to conferring trust is higher when there is a misalignment of cultural schemas. Trust can be conferred, for example, if the consumer believes an analysis that has an accompanying statistical rationale has greater validitylxxi and by extension places greater confidence in the analysis even if the consumer doesn't understand the statistics or their representation. This trust is conferred because of a broader cultural value and belief in the efficacy of scientific methods, including statistical measurement and analysis. Conversely, the quality of intelligence analysis and reports that lack statistical analysis are perceived by consumers as being lower, which extends to the analysts being perceived as less knowledgeable
39
Saab & McGill
and less trustworthy.lxxii In other words, despite the lack of schema overlap, trust is given because of authoritative reasons of expertise believed to have been applied to the intelligence. And when such authoritative expertise is not present, the trust in the analysis as well as the analyst is less readily conferred. Hindsight evaluation of probabilistic forecasts may deem such authoritative forecasts to be wrong, leading to a revocation of trust and the reluctance of analysts to include statistical measures in their forecasts.lxxiii The distinction between trust and confidence is difficult to make. Conferring either is often a matter of schematic (i.e., automatic, abductive, heuristic) processing of informationwe dont necessarily engage in a protracted internal dialogue in which we articulate all possible scenarios. Caroline Wesson argues that we often use a confidence heuristiclxxiv as an indicator of the validity of information we are given, especially when we are uncertain of the information and the choices we may make. People who rely more on a confidence heuristic to make choices are more resistant to persuasion when possessing prior information. We might say they are schematically rigid. They are less so when lacking prior information. The confidence heuristic is one in which, when faced with uncertainty, people operate a confidence heuristic rule, whereby they agree with the most confident advice available.lxxv Using our emergent cultural framework, contra Wesson, its not so much the confidence that the analyst attaches to the work, but the trust that the decision maker confers based on how he associates credibility of whats being said through confirmation biases, shared cultural understanding, or membership or reputation in a social network. Trust also reveals that schemas have an emotional componentin cases where confidence is misaligned, trust may be conferred for reasons of strengthening social network bonds. For example, I might not have any confidence at all in the advice being given by
40
Saab & McGill
someone. The advice, for example, might conflict sharply with my own understanding of the situation and potential courses of action. But because the advice giver is part of my extended social network, and may have gained a reputation for expertise in a particular area for which I have none, I might confer trust upon him even though our rationales conflict sharply, and I can provide no logical reason related to the intelligence to convey such trust. I might know in my gut that trusting them is the right thing to do despite any misgivings I may have. By trusting them, I might be signaling to them and others within my social network that my dependence upon them reinforces our shared cultural identity. If they are not already part of my social network, by bringing them into my social network, and by establishing and inclusion boundary around them, it provides the opportunity to develop a stronger shared cultural identity. This, in turn, leads to greater schema overlap as part of the self-reinforcing discourse of sense making and meaning making. The strengthening of social network bonds and the development of cultural schemas may arise because of circumstances where one is forced to cooperate, e.g., in an enemy of my enemy is my friend situation. I might need to extend trust to them for practical reasons or because of shared goals or motivations. In circumstances of forced trust, I might also be vigilant about potential betrayal and have a simultaneous distrust of my friend of convenience.
SUMMARY
Understanding the reasoning about and conferring of confidence levels by intelligence consumers is a cultural activity. We used an emergent culture framework, with schemas as a central mechanism, to explore this cultural activity rather than a classical notion of culture comprised of a set of norms, values and beliefs. This emergent framework allowed us to explore an individuals role in cultural activity without engaging in an ecological fallacy. It allowed us to
41
Saab & McGill
develop a dynamic model of sociocultural factors and describe their influence on an intelligence consumers reasoning and conferring of confidence in intelligence analyses. We showed that schemas are also integral to sense making and meaning making, which are entwined processes that occur at individual and sociocultural levels, respectively. We argued that the alignment of confidence levels among producers and consumers of intelligence was the product of a cultural reasoning process guided by shared schemas. We examined the sociocultural factors in reasoning in more detail. Presentation of information is complemented by the cultural factor of assessing the analysts understanding of the issue. It prompts a hermeneutic engagement with the consumer to determine how divergent the analysts schemas are from the consumers understanding. Social networks were described as the canvas through which our cultural backgrounds emerge. Cultural background can mean identity, expert judgment, or ways of thinking. Identity connects the individual to the social network. Expert judgment draws on the extended experience of the individual and his development of sophisticated schema patterns that he can bring to bear for a narrowed topic of interest. Ways of thinking refers to the schematizing of sense making and meaning making processes that connect knowledge patterns and data patterns. Our schematized ways of thinking tend toward a confirmation bias as we seek out salient elements of familiar patterns to help guide our expectations and understanding, and comport with our motivations and goals. Trust arises from a complex interplay of experience, social identity and schematic understanding. Using our emergent cultural framework, its not so much the confidence that the analyst attaches to the work, but the trust that the decision maker confers based on how he associates credibility of whats being saidthrough confirmation biases, shared cultural understanding, or membership or reputation in a social network. Our
42
Saab & McGill
cultural analysis of trust suggests that it, rather than confidence, should be the end-state for which producers and consumers of intelligence strive.
ACKNOWLEDGEMENTS
This work is supported by the National Geospatial-Intelligence Agency (under Grant No. MH1582-09-1-0020).
REFERENCES
i
Robert Jervis, "Why Intelligence and Policymakers Clash," Political Science Quarterly, Vol. 125, 2010, pp. 185-204. ii Ibid. iii See Raymond S. Nickerson, "Confirmation bias: A ubiquitous phenomenon in many guises," Review of General Psychology, Vol. 2, 1998, pp. 175-220, for discussion of beliefs, expectations, and hypotheses; and see Karl R. Popper, The Logic of Scientific Discovery. (New York: Basic Books, 1959); and Peter C. Wason, "On the failure to eliminate hypotheses in a conceptual task," Quarterly Journal of Experimental Psychology, Vol. 12, 1960, pp. 129-40, for a discussion of the historical roots of theories of confirmation bias. iv Susan G. Strauss, Andrew M. Parker and James B. Bruce, "The Group Matters: A Review of Processes and Outcomes in Intelligence Analysis," Group Dynamics: Theory, Research, and Practice, Vol. 15, 2011, p. 128. v J. Richard Hackman and Michael O'Connor, "What makes for a great analytic team? Individual vs. Team approaches to intelligence analysis," (Washington, DC: Intelligence Science Board, Office of the Director of Central Intelligence, 2004). vi David T. Moore, Sensemaking: A Structure for an Intelligence Revolution. (Washington, DC: National Defense Intelligence College Press, 2011); and Horst W.J. Rittel and Melvin M. Webber, "Dilemmas in a General Theory of Planning," Policy Sciences, Vol. 4, 1973, pp. 15569. vii "A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis," (US Government, 2009). viii Jeannie L. Johnson and Matthew T. Berrett, "Cultural Topography: A New Research Method for Intelligence Analysis," Studies in Intelligence, Vol. 55, 2011, p. 3. ix Claudia Strauss and Naomi Quinn, A cognitive theory of cultural meaning. (Cambridge: Cambridge University Press, 1997), p. 6. x Roy G. D'Andrade, The Development of Cognitive Anthropology. (Cambridge: Cambridge University Press, 1995), p. 140. xi Paul DiMaggio, "Culture and cognition," Annual Review of Sociology, Vol. 23, 1997, pp. 26388. xii Gary A. Klein and Caroline E. Zsambok, Naturalistic Decision Making. (Mahwah, NJ: Erlbaum Associates, Inc., 1997).
43
Saab & McGill
xiii
Gheorghe Tecuci, Mihai Boicu, David Schum and Dorin Marcu, "Overcoming Intelligence Analysis Complexity with Cognitive Assistants," (Learning Agents Center, George Mason University, 2009). xiv Claudia Strauss and Naomi Quinn, A cognitive theory of cultural meaning, p. 6. xv Usually referred to as mental models in the IC literature. See Richards J. Heuer, Psychology of Intelligence Analysis, (Bernan Assoc, 1999); Robert Jervis, Why Intelligence and Policymakers Clash; and Caroline J. Wesson, "The Communication and Influence of Confidence and Uncertainty," PhD Dissertation (Wolverhampton, UK: University of Wolverhampton, 2005), 256pp. xvi A Tradecraft Primer. xvii David T. Moore, Sensemaking: A Structure for an Intelligence Revolution. (Washington, DC: National Defense Intelligence College Press, 2011). xviii Ibid., p. x. xix Ibid., p. xxxv. xx Gary Klein, Brian Moon and Robert R. Hoffman, "Making Sense of Sensemaking 1: Alternative Perspectives," IEEE Intelligent Systems, Vol. 21, 2006, p. 71. xxi For a more thorough exposition of the phenomenological characterization in Figure 2, see David J. Saab and Uwe V. Riss, "Information as Ontologization," Journal of the American Society for Information Science and Technology, Vol. 62, 2011, pp. 2236-46. xxii Klein et al., Making Sense of Sensemaking. xxiii Fred Dretske, Knowledge and the Flow of Information. (Cambridge: The MIT Press, 1981). xxiv Michael Polanyi, Personal Knowledge. (Chicago: The University of Chicago Press, 1962). xxv Lev Semenovich Vygotsky, Thought and Language. (Cambridge, MA: The MIT Press, 1986). xxvi Johnson and Berrett, Cultural Topography, pp. 2-3. xxvii Nikolas Kompridis, "So We Need Something Else for Reason to Mean," International Journal of Philosophical Studies, Vol. 8, 2000, pp. 271-95. xxviii Abduction, analogical, cause-and-effect, comparative, conditional, criteria, decompositional, deductive, exemplar, inductive, modal logic, pros-vs-cons, residue, set-based, systemic, syllogistic, traditional logic, (see http://changingminds.org/disciplines/argument/types_reasoning/types_reasoning.htm). xxix Thomas S. Kuhn, The Structure of Scientific Revolutions. (Chicago: University of Chicago Press, 1996). xxx Johnson and Berrett, Cultural Topography. xxxi Tecuci et al., Overcoming Intelligence Analysis Complexity, p. 4. xxxii Johnson and Berrett, Cultural Topography, p. 3. xxxiii Barak A. Salmoni and Paula Holmes-Eber, Operational Culture for the Warfighter: Principles and Applications. (Quantico, VA: Marine Corps University Press, 2008), p. 13. xxxiv Jeffrey R. Cooper, "Curing Analytic Pathologies," (Central Intelligence Agency, 2005). xxxv See David V. Budescu, Stephen Broomell and Han-Hui Por, "Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change," Psychological Science, Vol. 20, 2009, pp. 299-308, for a discussion of how the imprecision of terms provides an illusion of communication; and see David V. Budescu and Thomas S. Wallsten, "Consistency in interpretation of probabilistic phrases," Organizational Behavior and Human Decision Processes, Vol. 36, 1985, pp. 391-405; regarding the assumption of consistency and similarity.
44
Saab & McGill
xxxvi
Nathan F. Dieckmann, "Communicating Risk in Intelligence Forecasts: The Consumer's Perspective," PhD Dissertation, (Eugene, OR: University of Oregon, 2007), p. 118. xxxvii Ibid, p. 120. xxxviii Charles Weiss, "Communicating Uncertainty in Intelligence and Other Professions," International Journal of Intelligence and CounterIntelligence 21 (2007): 57-85. xxxix Stephen Broomell and David V. Budescu, "Why are experts correlated? Decomposing correlations between judges," Psychometrika, Vol. 74, 2009, pp. 531-53. xl Vygotsky, Thought and Language. xli Heuer, Psychology of Intelligence Analysis. xlii Mentioned in A Tradecraft Primer, and Tecuci, et al., Overcoming Intelligence Analysis Complexity. xliii Dieckmann, Communicating Risk in Intelligence Forecasts, op. cit.; and Nathan F. Dieckmann, Robert Mauro and Paul Slovic, "The Effects of Presenting Imprecise Probabilities in Intelligence Forecasts," Risk Analysis, Vol. 30, 2010, pp. 987-1001. xliv Budescu et al, Improving Communication of Uncertainty. xlv David J. Saab, "A conceptual investigation of the ontological commensurability of spatial data infrastructures among different cultures," Earth Science Informatics, Special Issue on Spatial Data Infrastructures for the Sustainability of the Brazilian Amazon: Integrating People, Information, and Models, Vol. 2, 2009, pp. 283-297. xlvi Budescu et al., Improving Communication of Uncertainty; and Budescu and Wallsten, Consistency in interpretation of probabilistic phrases. xlvii Dieckmann, Communicating Risk in Intelligence Forecasts. xlviii Vicki M. Bier, "On the state of the art: risk communication to decision-makers," Reliability Engineering & System Safety, Vol. 71, 2001, pp. 151-7; and Budescu et al., Improving Communication of Uncertainty; and Dieckman, Communicating Risk in Intelligence Forcasts. xlix Timothy van Gelder, "Mindfulness Versus Metacognition, and Critical Thinking," in Tim van Gelder, Bringing visual clarity to complex issues, 2009, http://timvangelder.com/2009/05/27/mindfulness-versus-metacognition-and-critical-thinking/. l Dieckmann et al., The Effects of Presenting Imprecise Probabilities in Intelligence Forecasts. li Shawn P. Curley and J. Frank Yates, "The center and range of the probability interval as factors affecting ambiguity preferences," Organizational Behavior and Human Decision Processes, Vol. 36, 1985, pp. 273-87; and Hillel J. Einhorn and Robin M. Hogarth, "Ambiguity and uncertainty in probabilistic inference," Psychology Review, Vol. 92, 1985, pp. 433-61; and Peter Gardenfors and Nils-Eric Sahlin, "Decision making with unreliable probabilities," British Journal of Mathematical and Statistical Psychology, Vol. 36, 1983, pp. 240-51; and William D. Rowe, "Understanding uncertainty," Risk Analysis, Vol. 14, 1994, pp. 743-50. lii Dieckmann et al., The Effects of Presenting Imprecise Probabilities in Intelligence Forecasts. liii Jervis, Why Intelligence and Policymakers Clash. liv Jervis, Why Intelligence and Policymakers Clash, p. 199. lv Charles Weiss, Communicating Uncertainty in Intelligence and Other Professions. lvi Heuer, Psychology of Intelligence Analysis. lvii Johnson and Berrett, Cultural Topography. lviii Ibid, p. 3. lix Jervis, Why Intelligence and Policymakers Clash. lx Ibid, p. 187.
45
Saab & McGill
lxi
Bruce D. Berkowitz and Allan E. Goodman, Best Truth: Intelligence in the Information Age. (New Haven, CT: Yale University Press, 2000); and Thomas H. Hammond, "Intelligence Organizations and the Organization of Intelligence," International Journal of Intelligence and CounterIntelligence, Vol. 23, 2010, pp. 680-724. lxii Thomas H. Hammond, "Intelligence Organizations and the Organization of Intelligence," International Journal of Intelligence and CounterIntelligence, Vol. 23, 2010, pp. 680-724. lxiii B. Chandrasekaran, Joseph R. Josephson and V. Richard Benjamins, "What are ontologies, and why do we need them?" Intelligent Systems and Their Applications, IEEE, Vol. 14, 1999, pp. 20-26; and Thomas Gruber, "Toward principles for the design of ontologies used for knowledge sharing," International Journal of Human-Computer Studies, Vol. 43, 1995, pp. 907-28; and Michael Uschold, "Where are the semantics in the semantic web?" AI Magazine, Vol. 24, 2003, pp. 25-36; and Michael Uschold and Michael Grninger, "Ontologies: principles, methods, and applications," Knowledge Engineering Review, Vol. 11, 1996, pp. 93-155. lxiv David J. Saab, "Conceptualizing Space: Mapping Schemas as Meaningful Representations," in Intercultural Relations, M.A. Thesis (Cambridge, MA: Lesley University, 2003), 90pp, http://www.djsaab.info/thesis/djsaab_thesis.pdf; and Saab, A conceptual investigation; and Saab and Riss, Information as Ontologization; and David J. Saab and Frederico Fonseca, "Ontological Complexity and Human Culture," in Philosophy's Relevance in Information Science (Paderborn, Germany, 2008). lxv Nurit Gal-Oz, "TRIC: An Infrastructure for Trust and Reputation Across Virtual Communities," in Fifth International Conference on Internet and Web Applications and Services, May 9-15, 2010, ed. Tal Grinshpoun, Ehud Gudes and Ingo Friese (Barcelona, Spain: IEEE, 2010), 43-50, emphasis ours. lxvi Hammond, Intelligence Organizations and the Organization of Intelligence. lxvii Jervis, Why Intelligence and Policymakers Clash. lxviii Ibid. lxix Shradha Tibrewal and John Poertner, "Confidence and uncertainty in casework decisions: The supervisors role," 2000. lxx Wesson, The Communication and Influence of Confidence and Uncertainty, p. 215. lxxi Dieckmann, Communicating Risk in Intelligence Forecasts; and Dieckmann et al., The Effects of Presenting Imprecise Probabilities in Intelligence Forecasts. lxxii Dieckmann, Communicating Risk in Intelligence Forecast. lxxiii Ibid. lxxiv Wesson, The Communication and Influence of Confidence and Uncertainty. lxxv Ibid., p. 96.
46

Sociocultural Factors in The Intelligence Consumer's Reasoning About Analytic Confidence

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Sociocultural Factors in The Intelligence Consumer's Reasoning About Analytic Confidence

Caricato da

Copyright:

Formati disponibili

Sociocultural factors in the intelligence consumers reasoning about analytic confidence

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

SOCIOCULTURAL FACTORS IN REASONING

Saab & McGill

Sociocultural Factors in Reasoning

Figure 1. Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Figure 2. Ontologization: The Phenomenological Character of Information

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

IMPACT OF SOCIOCULTURAL FACTORS

PRESENTATION OF INFORMATION AND ANALYSIS

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

ASSESSING THE ANALYSTS UNDERSTANDING

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning

Saab & McGill

Sociocultural Factors in Reasoning