Validity Concepts in Research: an Integrative Approach

David Brinberg, University of Maryland ABSTRACT - The research process is described as the interrelationships of conceptual, methodological, and substantive domains. The conceptual domain is seen as concepts and ideas in abstract form; the methodological domain as the designs, strategies, measuring devices, and analytic techniques used to study a phenomenon or theory; and the substantive domain as the events/processes studied. The specification of each domain and their interrelations will be discussed. Traditional forms of validity will be presented within the proposed framework and new forms of validity researchers need to consider will also be derived from this framework
[ to cite ]:

David Brinberg (1982) ,"Validity Concepts in Research: an Integrative Approach", in NA - Advances in Consumer Research Volume 09, eds. Andrew Mitchell, Ann Abor, MI : Association for Consumer Research, Pages: 40-44.

Advances in Consumer Research Volume 9, 1982

Pages 40-44

VALIDITY CONCEPTS IN RESEARCH: AN INTEGRATIVE APPROACH David Brinberg, University of Maryland ABSTRACT The research process is described as the interrelationships of conceptual, methodological, and substantive domains. The conceptual domain is seen as concepts and ideas in abstract form; the methodological domain as the designs, strategies, measuring devices, and analytic techniques used to study a phenomenon or theory; and the substantive domain as the events/processes studied. The specification of each domain and their interrelations will be discussed. Traditional forms of validity will be presented within the proposed framework and new forms of validity researchers need to consider will also be derived from this framework INTRODUCTION As the reader is aware, the concept of validity has taken many forms and meanings to describe different components of the research process. Traditionally, the research process has been divided into four basic components: (l) the specification, measurement and/or manipulation of a theoretical variable(s), (2) the selection of a particular type of research design, (3) the analysis of the data obtained from the study and (4) the interpretation and robustness of the findings. Associated with each component of the research process are one or more forms and/or meanings of validity to assess the "adequacy" of the particular research component. For instance, construct validity, convergent, and discriminant validity for theoretical variables; internal validity for research design; statistical conclusion validity for data analysis; and external validity for the robustness of research findings. Give the current state of the art concerning the various forms and meanings of validity issues, several options seem available to researchers. One option would involve further refinement and specification of current validity concepts. By selecting this option, researchers would assume that the basic formulations of the validity issues are acceptable and that research is needed to further articulate the validity concepts. A second option is to invent new forms of validity to deal with methodological and theoretical developments. Researchers selecting this option would assume current forms of validity are inadequate to deal with new developments. Thus, new forms of validity would be needed. A third option would be: (a) to develop a conceptual scheme that explicitly describes the components of the research process and their interrelationships, and (b) to use this scheme to "locate" or derive traditional forms of validity (e.g., construct validity, external validity) and identify new validity issues that researchers need to consider. A researcher selecting this option would assume that a conceptual scheme would help clarify the research process, the various

roles of validity in research as well as the relationships among the validity concepts. Finally, a fourth option is to accept the current state of the arc and conduct no further research. The option to be selected and expanded on in this paper is the development of a conceptual scheme: (1) that describes the research process and (2) is used to derive and locate various forms and meanings of validity. THE RESEARCH PROCESS Research involves drawing upon elements and relations from three basic domains: (a) a conceptual domain, which includes concepts and relations considered in abstract form, (b) a methodological domain, which includes instruments and techniques for obtaining observations and for relating sets of observations; and (c) a substantive domain, which includes events, processes, and phenomenon in the "real" world. Any research project must contain elements and relations from each of these domains. Thus, it is not possible to conduct research, without some method, some concept (or set of concepts), and some event or process. Elements and relations from each of these domains are not all combined simultaneously. Research generally proceeds by combining two of the domains, to form some structure, and subsequently incorporating (i.e., integrating) the third domain with the developed structure. With three domains, there are at least three patterns for combining the domains. Those three ways represent three distinct research paths; and they pose different advantages and limitations for the investigator. First, one can combine elements and relations from the conceptual domain with techniques for obtaining observations and relating these observations--without yet bringing in events and processes from the third (substantive) domain. This structure is a plan for doing research, that is, a research design but is not a completed research project until it has been integrated with the substantive domain. This structure will yield a set of intended measures and a plan for relating (or comparing) these measures. For instance, a researcher may measure an attitude (i.e., an element from the conceptual domain) by using a semantic differential scale. In addition, a researcher may be interested in examining the relations among a set of concepts (e.g., attitudes, norms, and intention) by selecting a particular type of design structure (e.g., within subject factorial design). It is important to note the combination of these two domains is not a completed research study until it has been integrated with a particular event or process (i.e., elements and relations from the substantive domain). Alternatively, a researcher may combine elements and relations from the conceptual domain with events and processes from the substantive domain. The outcome of this structuring process may reasonably be called a theory. The mapping of an element (i.e., single concept) from the conceptual domain with an event from the substantive domain may be termed a theoretical construct. The mapping of a set of constructs with a process or phenomenon may he termed a hypothesis. For instance, a researcher may combine the concept of salience with information retrieval (which is the particular substantive event) in order to form a theoretical construct. If a researcher is interested in a particular process, for instance, a person's decision to purchase generic prescription drugs, the mapping of the concepts of norms, attitudes, and self-concept to predict this process may be termed a hypothesis. This theory is not yet a completed research study until it is integrated with techniques and strategies from the methodological domain. A third pattern is to combine elements and relations from the methodological domain with events and processes from the substantive domain. Such activity involves mapping methods for making observations onto observable events, and mapping methods for relating (segregating, aggregating, contrasting) sets of observations or events. We can designate such method-substance mapping as constructing a body of data. For instance, a researcher may use concurrent protocol techniques to observe an individual's decision process. However, this body of empirical information is not a completed research study until it has been connected, that is, construed in terms of, elements and relations from the conceptual domain. To summarize, given the three domains, three research structures may be developed--a design formed by the combination of the conceptual and methodological domains; a theory formed by the combination of the conceptual

and substantive domains; and a body of data formed by the methodological and substantive domains. Each of these research structures are not a completed study until they have incorporated the third domain. One path a researcher may follow is to connect events and/or processes from the substantive domain with a particular design structure. This path (labelled as Path A in Figure l) is an implementation process. For instance, a researcher may use a Thurstone scale to measure attitude but to implement this scale, it is necessary to select a particular event (e.g., attitudes toward generic drugs). Another researcher may be interested in examining the relations among attitude, norms and intention and use a within subject factorial design. Implementing this design would require the researcher to connect it with events and/or processes from the substantive- domain, (e.g., the purchase of generic drugs; the use of birth control pills). An alternative research path is to build a theory (i.e., develop a set of constructs and some hypothesis) and test this theory by selecting a particular method. This path (labelled as Path 8 in Figure 1) may be seen as testing a theory. For instance, a researcher may use a theoretical construct of salience and use reaction time technology to test this construct. Another researcher may have a hypothesis concerning a particuLar process (e.g., information overload) and select a method (e.g., behavioral process technology) to Lest this hypothesis. A third path's researcher may take is to construct a body of data and subsequently select concepts to explain these data. This path (labelled as Path C in Figure 1) may be seen as an interpretation or explanation process. For instance, a researcher may have collected a body of data using concurrent protocol techniques and then attempt to interpret the data by selecting concepts to categorize various components of the data. To summarize, there are three paths a researcher may follow: implementing a research design; testing a theory; explaining a body of data. In one sense, these paths are all the same in that they involve elements and relations from each of the three domains. In another sense, they are strived at by 3 different sequence of steps and will reflect the orientation (style) of a particular researcher or an area of research. Adhering to any single path will have certain advantages---but will also have certain limitations---as with any mono-strategy approach. Figure 1 summarizes the relations among the domains, structures, and research paths. In addition to these two steps of structuring and integrating the three domains, there are activities that occur prior to as well as subsequent to the conduct of the research study that are important to the research process. These two stages will be referred to as pre-study and post-study activities respectively. Pre-study activities Using the scheme presented in Figure 1, the structuring and integrating steps of the research process presuppose a selection (sampling) of elements and relations from the three domains. Elements and relations from the conceptual domain consists of concepts and patterns (conceptual relations) among such concepts. In the substantive domain, elements are observable events and relations are sets of events or processes. Finally, in the methodological domain, elements are methods/strategies for making observations and relations are techniques for making comparisons among a set of observed events or concepts. FIGURE 1 However, such selection (sampling) from the domains implies that there has been some prior exploration/ analysis/understanding of those domains and the kinds of elements and relations that can be drawn from them. Such explorations are necessary preconditions for conducting a research project but are generally not considered part of the research process. For instance, in the methodoLogical domain, this prestudy activity leads to the development and use of specific cools for making observations (e.g., behavioral process technology) and for comparing the relations among concepts or events (e.g., three mode factor analysis). Similar activities are conducted in both the conceptual and methodological domains.

Post-study research activities After completion of one of the research paths previously presented, there are several crucial activities before a study can become "useful knowledge." These post-study activities are different for the three different research oaths . For research path A, the researcher combines the conceptual and methodological domains to form a research design and subsequently combines this structure with the substantive domain. When the researcher has completed this process, they are still faced with a key problem: the extent to which the findings will generalize to or across other substantive events. This is the problem of the robustness of findings within the substantive domain. For research path B, the researcher combines the conceptual and substantive domains to form 3 theory and subsequently tests this theory by selecting a particular method. Upon completion of this process, the researcher is still faced with a key problem: the extent to which the findings will generalize across other methods. This is the problem of robustness within the methodological domain. For research path C, the researcher combines the methodological and substantive domains to form a body of data and subsequently attempts to explain these data by selecting a set of concepts. Upon completion of this process, the researcher is still faced with a key problem: the extent to which the findings will stand up when compared with other elements and relations from the conceptual domain. This is the problem of the robustness of the concepts used to explain the body of data. All three post-research activities deal with the adequacy of the elements and relations sampled from the three domains--and all three are variations of the general concept of external validity. These various forms of external validity will be discussed in more detail subsequently. VARIOUS MEANINGS OF VALIDITY What I hope to establish here is a systematic ordering of different forms and meanings of validity, in relation to one another and in relation to parts of the research process, so that the full power of those concepts, and the relations among them may be more clearly seen. Associated with each stage of the research process is a different fundamental meaning of validity. In Stage 1, validity cakes on the meaning of value. By this, we mean the importance/relevance/truth of the concepts, methods, and substance selected for study. A second meaning --validity as correspondence --- is associated with Stage 2 of the research process. Many of the traditional forms of validity hinge on the correspondence between two "entities." For instance, construct validity deals with the correspondence between a theoretical concept and a particular event; convergent validity with the correspondence of a concept with two or more methods to measure the concept. The different forms of validity, internal to the research study, are organized in terms of two categories: (1) validities associated with the structures formed from the combination of elements and relations from two of the domains and (2) validities associated with the integration of the structure with elements and relations from the third domain. A third meaning --- validity as robustness, generalizability, dependability --- is associated with Stage 3 of the research process. The primary focus with this meaning of validity is determining the scope of a set of findings. As will be discussed, the particular scope a researcher is concerned with varies as a function of the research path selected in ScaRe 2. Stage l: Validity 35 Value Several researchers (e.g., Kaplan, 1964; Gergen, 1973; Kuhn, 1962) have discussed the influence of values on research. These values guide the selection of standards to be used in a particular research area. This meaning of validity --- as value ---- is used to identify the concepts to study; the methods to use; and the events/processes to examine. Generally, the values used in the selection process are determined by the prevailing paradigm. As Kuhn (1962) has noted, when paradigms change, values will also change. Given the conceptual scheme Presented earlier, "paradigm shifts" may be regarded as changes in the values involved in selecting elements and relations from each of the three domains.

There are many values associated with each domain. For instance, values generally used in the conceptual domain are: testable, quantifiable, robust, internally consistent; in the methodological domain: significance testing, accuracy, repeatable, quantifiable; in the substantive domain: observable, what is "real." The concepts, methods, and events selected for study have changed over time --- in response to the changing values. For instance, in the earlier 1900's, introspection was an acceptable technique for the study of human learning (decision making). When Watsonian behaviorism developed, the technique of introspection was discarded because it would not yield "valid" (i.e., valuable) data. Note that a shift in what was considered valuable resulted in the elimination of a particular method. Recently, concurrent protocol analysis has been developed as 3 technique for studying decision making and many aspects of this technique are similar to introspection. Again, a shift in values has resulted in the use of an "acceptable" technique to yield "valid" data. A similar effect of values may be found in the conceptual and substantive domains. For instance, early in this century, an instinct was a valid concept to select. However, given the development of the logical positivist philosophy, concepts were considered of value if they were quantifiable. Since no quantifiable index of instinct was developed, this concept was no longer considered valid. Finally, in the substantive domain, events considered "real" will also change as values change. Prior to the Freudian revolution, unconscious forces were not considered valid (i.e., real). For research within a Freudian perspective, unconscious forces are "real" and valid events. However, within much of the traditional information processing literature, unconscious forces are not considered "real" (i.e., they are not valid events). Table 1 contains an example of a concept, method, and event considered acceptable (and unacceptable) within the information processing area. The reader should note that in other areas (e.g., astrological science), the concept, method, and event listed may be seen as valid (i.e., valuable). TABLE 1 STAGE I. PRIOR VALIDITIES Stage 2: Validity as Correspondence The second stage of the research process consists of three alternative paths, each with two major steps. As mentioned earlier, the first step involves combining the three domains pairwise to form a structure --- design, theory, body of data. Since elements and relations are selected for each structure, there are six different forms of validity to consider. This set of six validities may reasonably be called "logical validities" since the threats to these forms of validity occur independent of the completion of the research study (i.e., integration of the structure with the third domain). These validity issues can be evaluated prior -to conduct of the research study. The second set of forms of validity are associated with the integration step in Stage 2. Since elements and relations from the third domain will be integrated with the structure formed in step l, six different forms of validity need to be considered. The potential threats to these forms of validity occur during the actual conduct of the research study. All 12 of these forms of validity might reasonably be called "internal validities" since they are internal to the ongoing research process within a study. Table 2 contains a list of the various forms of validity associated with each structure as well as with each integration step. Some of the traditional forms of validity will be presented to highlight the association between the validity issues and the components of the research process. Space limitations preclude a detailed description of each form of validity and the associated validity threats. A more detailed discussion of these issues may be found in Brinberg and McGrath (in press). TABLE 2 STAGE II. INTERNAL VALIDITIES The matching of concepts and methods at the elements level (i.e., a design structure) yield a set of intended measures. The correspondence between the methods and concepts may simply be termed instrument validity, that is, the extent to which a measure can correspond with a concept or set of concepts. One major threat to the validity of

the instrument is the potential confounding of instrument and concept. Campbell and Fiske (1959) describe a multitrait-multimethod approach that attempts to identify the degree to which there is instrument-concept confounding. Combining concepts and methods at the relations level deaLs with the correspondence between the design structure selected and the relations among the concepts. Traditionally, this form of validity has been called internal validity (Campbell and Stanley, 1966). However, since we have used the term internal validity to refer to all the forms of validity internal to the research process, the term comparison validity has been developed to replace the traditional term of internal validity. The central notion with this form of validity deals with the extent to which planned comparisons within the study will permit clear inferences about the relations among the concepts. The threats to comparison validity include: history, maturation, testing, as well as misspecification of the research design (e.g., Judd and Kenny, in press). The other forms of validity within the "logical validities" all deal with correspondence between an element/ relation from one domain with an element/relation from the second domain. Where appropriate, traditional forms of validity have been included within this conceptual scheme. However, the definition of the form of validity is derived from the meaning of validity within this step and stage of the research process, that is, the degree of correspondence between elements and relations selected from two of the three domains. The second step of each of the three paths involves connecting elements and relations from the third domain with the structure formed in the first step. These "integration" validities involve the correspondence between elements and relations selected from the third domain with the structure formed in step 1. For path A, when a design structure is connected with processes from the substantive domain, the correspondence deals with the execution of the study. This form of validity may reasonably be called execution validity. Cook and Campbell (1979) discuss threats to this form of validity that deal with imperfections in conducting (i.e., executing) the study. For example, demand characteristics and experimenter biases i.e., cues the subject attends to that are incidental to the main emphasis of the research, will influence the correspondence between the design and the process selected from the substantive domain. This "artifact" might potentially threaten the validity of the execution of the design. In addition, the orientation the subject takes to a particular instrument (e.g., response sets) may influence the validity of the instrument's use. This form of validity may reasonably be called instrument use validity. It.differs from instrument validity since the potential threats to the use of the instrument only occur when the instrument is used. The other forms of validity within the integration step of Stage 2 also deal with the correspondence between the structure formed in step l and the third domain. The reader should note that the validities associated with the integration step of the research process deal with issues that arise during the conduct of the study --- not prior to the study. A more detailed discussion of these forms of validity may be found in Brinberg and McGrath (in Dress). Stage 3: Validity as Robustness Given the completion of the structuring and integration steps in Stage 2, there is still a need to assess the robustness (i.e., scope) of the findings. Traditionally, the robustness of research findings deal with the external validity of the research conclusions. Since three different paths may be pursued in Stage 2, three different external validity issues may arise. If a researcher develops a design, and subsequently implements this design by selecting substantive events and processes, they are concerned with the extent to which the findings will generalize across other substantive events. This form of validity may reasonably be termed ecological validity. For instance, a researcher may use a particular design (e.g., within subject factorial design to study the relations among intention, attitude, and norms) and select an event (e.g., purchasing clothes) to implement this design. Generally, the researcher is concerned with the extent to which their findings will be robust (i.e., invariant) across other events selected from the substantive domain (e.g., other behaviors, respondents, settings). If a researcher first forms a theory and subsequently selects a method to test this theory, they are generally concerned with the robustness of their findings across different methods. This form of external validity may

reasonably be called methodological validity. For instance, a researcher may select different methods --behavioral process technology and reaction time technology --- to test a theory of information overload. The concern with this form of validity is the extent to which the findings can be reproduced given two or more different methods. As with the ecological validity, this form of validity involves repeated sampling from the domain of interest -- in this case, the methodological domain. Finally, a researcher may collect a body of data and attempt to explain this structure by selecting a set of concepts. The extent to which a set of concepts is adequate to explain the body of data deals with the third form of external validity -- explanatory validity. For instance, a researcher may use concurrent protocol techniques to construct a body of data. The robustness involved in attempting to explain this body of data deals with the extent to which the category system (i.e., set of concepts) selected from the conceptual domain is adequate to explain the body of data. To assess the generalizability of a -particular category system, a researcher is likely to develop other category systems, that is, repeatedly sample from the conceptual domain, to determine which set of concepts provides the most accurate explanation of the body of data. Table 3 summarizes the three different forms of external validity. TABLE 3 STAGE III. EXTERNAL VALIDITIES IMPLICATIONS Historically, the validity concepts described in this paper seemed to have been developed ad hoc (i.e., without a theoretical foundation) to assess the adequacy of a new component of the research process that had recently been "discovered" (e.g., the recent development of statistical conclusion validity to deal with the adequacy of inferences from various data analytic techniques). A serious limitation of the cumulative work concerning the research process and the associated forms of validity has been the absence of a theoretical/conceptual organization and explanation of the interrelations among the components of the research process and with the various validity concepts. One potentially useful feature of the scheme presented in this paper is that the components of the research enterprise are explicitly specified and interrelated. This may help researchers come to a better understanding of the advantages and limitations of certain research "patterns." This conceptual scheme was also used to explicitly associate the forms and meanings of validity with specific components of research process. Rather than add new validities to describe the research process, the forms and meanings of validity were derived from the general scheme. This approach has the advantage of providing a theoretical basis for the different validity concepts. A third issue addressed was the use of this scheme to highlight similarities and distinctions among the validity concepts as well 35 identify new forms of validity researchers need to consider. This conceptual scheme was also used to further differentiate some forms of validity into several components (e.R., divide external validity into several components). Finally, I suggest that this framework can help in planning any given study by highlighting the specific validity issues most germane to and most problematic for that study. It can also help, I believe, in efforts to integrate and assess the work in a given research area, by pointing up which aspects of the overall validity question represent the most serious limitations of the cumulative work in that area.

Main Source: http://www.acrwebsite.org/volumes/v09/09042t01.gif

REFERENCES Brinberg, D. and McGrath, J.E. (in press), A network of validity concepts within the research process. In Brinberg, D. & Kidder, L. (Ed ;). New Directions for Methodology of Social and Behavioral Science: Forms of Validity in Research. Campbell, D.T. and Fiske, D.W. (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 30, 81-105. Campbell, D.T. and Stanley, J.C. (1966) Experimental and quasi-experimental designs for research. Chicago: RandMcNally. Cook, T.S. and Campbell, D.T. (1979) Design and analysis of quasi-experiments for field settings. Chicago: RandMcNally. Gergen, K.J. (1973) Social psychology as history. Journal of Personality and Social Psychology, 26, 309-320. Judd, C.M. and Kenny, D.A. (in DreSS). Research design and research validity. In Brinberg, D. & Kidder, L. (Eds). New Directions for Methodology of Social and Behavioral Science: Forms of Validity in Research. Kaplan, A. (1964) The conduct of inquiry. Scranton, Pa. Chandler Pub. Co. Kuhn, T.S. (1962) The structure of scientific revolutions Chicago: University of Chicago Press.