TOPIC PRESENT: VALIDITY AND RELIABILITY

Group members :
1) PhamThi Ngoc Bi
2) Nguyen Van Cuong 3) Le Thi Ha 4) Phan Thi Bich Hien 5) Nguyen Ngoc Minh Nguyet

OUTLINE
I. Introduction. II. Validity. 1. Validity in Qualitative Research. 1.1. Difinition. 1.2.Types of Validity in Qualitative Research. 1.3. Establishing Validity in Qualitative Research. 1.4.Threats to External Validity in Qualitative Research. 2. Validity in Quantitative Research 2.1. Definition. 2.2. Types of Validity in Quantitative Research. 2.3. Threats to Internal and External Validity. 3. Enssuring Validity.

OUTLINE
III. Reliability. 1. Reliability in Quantitative Research. 1.1. Definition. 1.2. Types of Reliability in Quantitative Research. 2. Reliability in Qualitative Research. 1.1. Definition. 1.2. Types of Reliability in Qualitative Research. IV. Validity and Reliability in data collection methods. V. Conclusion

I. Introduction
When we do a research, reliability and validity are the most two important criteria for assuring the quality of the data collection procedures. Reliability provides information on the extent to which the data collection procedure elicits accurate data while validity provides information on the extent to which the procedure really measures what it is supposed to measure. It is important to examine the quality of the procedure before it is administered in the actual research. The researcher examines the different data collection procedures in order to avoid problems during the administration of actual research. The research is invaluable if the data is unreliable and invalid. Thus, in this presentation we would like to concentrate on the validity and reliability in doing a research, particularly in quantitative and qualitative research.

II. Validity
1. Validity in Qualitative Research. 1.1.Types of Validity in Quatitative Research. - Descriptive - Interpretive, - Theoretical, - Generalizability, - Evaluative. 1.2. Establishing Validity in Qualitative Research. - Persistent observation - Triangulation - Peer debriefing - Negative case analysis. - Member checking . 1.3.Threats to External Validity in Qualitative - Research. - Selection effects, - Setting effects, - History effects, - Construct effects.

1.1. Definitions
Validity is of an assessment in the degree to which it measure what it is supposed to measure. (Kiwipedia) Validity refers to the extent to which the data collection procedure measures what it intends to measure. (Herbert W.Seliger Elena Shohamy)

1.2. Types of Validity in Quatitative Research.

Maxwell (1992) argues for five kinds of validity in qualitative methods that explore his notion of understanding: - Descriptive validity (the factual accuracy of the account, that it is not made up, selective, or distorted); in this respect validity subsumes reliability; it is akin to Blumenfeld-Joness (1995) notion of truth in research-what actually happened (objectively factual); - Interpretive validity (the ability of the research to catch the meaning, interpretations, terms, intentions that situations and events, i.e. data, have for the participants/subjects themselves, in their terms); it is akin to Blumenfeld-Joness (1995) notion of fidelitywhat it means to the researched person or group (subjectively meaningful); interpretive validity has no clear counterpart in experimental/positivist methodologies;

1.2. Types of Validity in Quatitative Research.

-Theoretical validity (the theoretical constructions that the researcher brings to the research (including those of the researched)); theory here is regarded as explanation. Theoretical validity is the extent to which the research explains phenomena; in this respect is it akin to construct validity (discussed below); in theoretical validity the constructs are those of all the participants; - Generalizability (the view that the theory generated may be useful in understanding other similar situations); generalizing here refers to generalizing within specific groups or communities, situations or circumstances validly) and, beyond, to specific outsider communities, situations or circumstance (external validity); internal validity has greater significance here than external validity;

1.2. Types of Validity in Quatitative Research.

- Evaluative validity (the application of an evaluative framework, judgemental of that which is being researched, rather than a descriptive, explanatory or interpretive one). Clearly this resonates with critical-theoretical perspectives, in that the researchers own evaluative agenda might intrude.

1.3. Establishing Validity in Qualitative Research.

Lincoln and Guba (1985: 219, 301) suggests that validity can be addressed by: - Prolonged engagement in the field; -Persistent observation (in order to establish the relevance of the characteristics for the focus); - Triangulation (of methods, sources, investigators and theories); - Peer debriefing (exposing oneself to a disinterested peer in a manner akin to cross-ex- Amination, in order to test honesty, working hypotheses and to identify the next steps in the research); - Negative case analysis (in order to establish a theory that fits every case, revising hypotheses retrospectively); - Member checking (respondent validation) to assess intentionality, to correct factual errors, to offer respondents the opportunity to add further information or to put information on record; to provide summaries and to check the adequacy of the analysis).

1.4.Threats to External Validity in Qualitative Research. - In naturalistic research threats to external validity include (Lincoln and Guba, 1985:189, 300): - Selection effects (where constructs selected in fact are only relevant to a certain group); - Setting effects (where the results are largely a function of their context); - History effects (where the situations have been arrived at by unique circumstances and, therefore, are not comparable); - Construct effects (where the constructs being used are peculiar to a certain group).

2. Validity in Quantitative Research

2.1.Definition. 2.2.Types of Validity in Quantitative Research. - External Validity. - Internal Validity. * Content Validity. * Construct Validity. * Criterion - Related Validity.

2. Validity in Quantitative Research

2.1. Definition. According to Joppe (2000) provides the explanation of what validity is in quanlitative research: Validity determines whether the research truly measures that which it was intended to measure or how truthful the research results are.

2.2. Types of Validity in Quanlitative Research.

- External Validity External validity addresses the ability to apply with the confidece the findings of the study to other people and other situations, and ensures that the conditions under which the study is carried out are representative of the situations and time to which the results are to apply.(Black 1999). The sample of participants drawn from the population of interest must be representative of the population at the time of the study. Finally, representative samples should be drawn with reference to relevant variables in study, such as gender and age.

2.2. Types of Validity in Quanlitative Research. - Internal Validity Internal Validity addresses the reasons for the outcomes of the study, and helps to reduce other, often unanticipated, reasons for these outcomes. Three approaches to assessing internal validity are content validity, construct validity and Criterion related validity (Eby 1993, Punch 1998)

2.2. Types of Validity in Quanlitative Research.

- Content Validity: To demonstrate this form of validity the instrument must show that it fairly and comprehensively covers the domain or items that it purports to cover.

Example : The researcher wished to see how well a group of students could spell 1,000 words in french but decided only to have a sample of fifty words for the spelling test, then that the test would have to ensure that it represented the range of spellings in the 1,000 words may be ensuring that the spelling rules had all been included or that posible spelling errors had been covered in the test in the proportions in which they occurred in the 1,000 words.

2.2. Types of Validity in Quanlitative Research.

- Construct Validity Construct validity involves demonstrating relationships between the concepts under study and the construct or theory that is relevant to them. For example, the group of drugs known as antibiotics (construct) comprises several distinct drugs (concepts, i.e. Penicillin, erythromycin and oxytetracyline) which are individual drugs but related to one another in that their primary purpose is to treat bacterial infection.

- Criterion - Related Validity

This form of validity endeavours to relate the results of one particular instrument to another external criterion. (Louis Cohen) When researchers are developing a test for obtaining information to distinguish between people who are field dependent and field independent, they will need to compare the instrument with another procedure which is acceptable as a valid measure of these variables. If the two procedures correlate with one another, that will provide evidence of the validity of the instrument. (Herbert W.Seliger Elana Shohamy)

3. ENSURING VALIDITY
3.1. Ensuring validity at the design stage, 3.2. Ensuring validity at the data collection stage, 3.3. Ensuring validity at the data analysis stage, 3.4. Ensuring validity at the data reporting stage.

3.1. Ensuring Validity at The Design Stage.

- Choosing an appropriate time scale; - Ensuring adequate resources for the research; - Selecting appropriate methodology; - Selecting appropriate instruments; - Using an appropriate sample; - Ensuring reliability; - Selecting appropriate foci; - Avoiding a biased choise of researcher or research team.

3.2. Ensuring Validity at the Data Collection Stage

- Reducing the Hawthorne effect. - Minimizing reactivity effects. - Avoiding drop-out rates amongst respondents. - Taking steps to avoid non-return of questionnaires. - Avoiding too long or too short an interval between pre-tests and post-tests. - Ensuring inter-rater reliability. - Matching control and experimental groups - Ensuring standardized procedures for gathering data - Building on the motivations of respondents - Tailoring the instruments to the concentration span of the respondents and addressing other situational factors. - Addressing researcher characteristics (attitude, gender, race, race, age, personality, dress....)

3.3. Ensuring Validity at the Data Analysis Stage

- Using respondent validation; - Avoiding subjective interpretation of data (e.g. being too generous or too ungenerous in the award of marks), i.e. lack of standardization and moderation of results; - Reducing the halo effect, where the researchers knowledge of the person or knowledge of other data about the person or situation exerts an influence on subsequent judgements; - Using appropriate statistical treatments for the level of data (e.g. avoiding applying tech- niques from interval scaling to ordinal data or using incorrect statistics for the type, size,complexity, sensitivity of data);

3.3. Ensuring Validity at the Data Analysis Stage

- Recognizing spurious correlations and extraneous factors which may be affecting the data (i.e. tunnel vision); - Avoiding poor coding of qualitative data; - Avoiding making inferences and generalizations beyond the capability of the data to support such statements; - Avoiding the equating of correlations and causes; - Avoiding selective use of data; - Avoiding unfair aggregation of data (particularly of frequency tables); - Avoiding unfair telescoping of data (degrading the data); - Avoiding Type I and/or Type II errors.

3.4. Ensuring Validity at The Data Reporting Stage.

- Avoiding using data very selectively and unrepresentatively (for example, accentuating the positive and neglecting or ignoring the negative); - Indicating the context and parameters of the research in the data collection and treatment, the degree of confidence which can be placed in the results, the degree of context-freedom or context-boundedness of the data (i.e. the level to which the results can be generalized); - Presenting the data without misrepresenting their message; - Making claims which are sustainable by the data; - Avoiding inaccurate or wrong reporting of data (i.e. technical errors or orthographic errors); - Ensuring that the research questions are answered; releasing research results neither too soon nor too late.

III.Reliability in Qualitative and Quantitative Research.

1.Reliability in Qualitative Research. 1.1. Definition 1.2. Types of Reliability in Qualitative Research. - Stability of observations. - Parallel form. - Inter - Rater Reliability. 2. Reliability in Quantitative Research. 2.1. Definition 2.2 Types of Reliability in Quantitative Research. - Reliability as Stability. - Reliability as Equivalence. - Reliability as Internal Consistency.

1. Reliability in Qualitative Research

1.1. Definition: - In qualitative research, reliability can be thought of as the trustworthiness of the procedures and data generated (Stiles 1993). - Reliability is concerned with the extent to which the results of a study or a measure are repeatable in different circumstances (Bryman 2001).

1. Reliability in Qualitative Research

1.2Types of reliability in qualitative research: Denzin and Lincoln (1994) suggest that reliability as replicability in qualitative research can be addressed in several ways: 1.2.1 Stability of observations: Whether the researcher would have made the same observations and interpretation of these if they had been observed at a different time or in a different place. 1.2.2 Parallel forms: Whether the researcher would have made the same observations and interpretation of what had been seen if she had paid attention to other phenomena during the observation 1.2.3 Inter- rater reliability: Whether another observer with the same theoretical framework and observing the same phenomena would have interpreted them in the same way.

2.Reliability in Quantitative Research

2.1. Definition:
- Joppe (2000) defines reliability as : .... The extent to which results are consistent over time and an accurate representation of the total population under study is referred to as reliability and if the results of a study can be reproduced under a similar methodology, then the research instrument is considered to be reliable. - Reliability is essentially a synonym for consistency and replicability over time, over instruments and over groups of respondents(Louis Cohen, Lawrence Manion and Keith Morrison 2000)

2.Reliability in Quantitative Research

2.2. Types of reliability in quantitative research: - There are three principal types of reliability: stability, equivalence and internal consistency 2.2.1 Reliability as stability ( test- retest): In this form reliability is a measure of consistency over time and over similar sample. A reliable instrument for a piece of research will yield similar data from similar respondents over time. 2.2.2 Reliability as equivalence: Within this type of reliability there are two main sorts of reliability : * Firstly, Reliability may be achieved, through using equivalent forms (alternative forms) of a test or data-gathering instrument. * Secondly, reliability as equivalence may be achieved through inter-rater reliability. 2.2.3 Reliability as internal consistency: Whereas the test/ retest method and the equivalence forms method of demonstrating reliability require the tests or instruments to be done twice, demonstrating internal consistency demands that the instruments or tests be run once only through the split- half method.

- The following table outlines three common reliability measures

Type of Reliability
Stability or Test-Retest

How to Measure Give the same assessment twice, separated by days, weeks, or months. Reliability is stated as the correlation between scores at Time 1 and Time 2. Create two forms of the same test (vary the items slightly). Reliability is stated as correlation between scores of Test 1 and Test 2. Compare one half of the test to the other half. Or, use methods such as Kuder-Richardson Formula 20 (KR20) or Cronbach's Alpha.

Equivalent Form (Alternate Form)

Internal Consistency

 The values for reliability co-efficient range from 0 to 1.0. A co-efficient of 0 means no reliability and 1.0 means perfect reliability. Since if the reliability of a standardized test is above .80, it is said to have very good reliability; if it is below .50, it would not be considered a very reliable test. Generally, the higher the co-efficient, the more reliable the procedure is.

III. VALIDITY AND RELIABILITY IN SOME INSTRUMENTS

1. VALIDITY AND RELIABILITY IN INTERVIEWS 2. VALIDITY AND RELIABILITY IN QUESTIONNAIRES 3. VALIDITY AND RELIABILITY IN EXPERIMENTS 4. VALIDITY AND RELIABILITY IN OBSERVATIONS 5. VALIDITY AND RELIABILITY IN TESTS 6. VALIDITY AND RELIABILITY IN LIFE HISTORIES

1. VALIDITY AND RELIABILITY IN INTERVIEWS

* Cause of invalidity: bias (a systematic or persistent tendency to make errors in the same direction, that is, to overstate or understate the "true value" of an attribute).
(Lansing, Ginsberge and Braaten, 1961)

=> To achieve greater validity -> minimize the amount of bias as much as possible.

1. VALIDITY AND RELIABILITY INTERVIEWS Causes of bias in interviewing:

- Biased sampling; - Poor rapport between interviewers and interviewee; - Changes to question wording; - Poor prompting and biased probing; - Poor use and management of support materials; - Alterations to the sequence of questions; - Inconsistent coding of responses; - Selective or interpreted recoding of data/ transcripts; - Poor handling of difficult interviews. (Oppenheim,1992)

1. VALIDITY AND RELIABILITY INTERVIEWS - Issue of leading questions. - Issue of power. - Problems in conducting telephone interviews

1. VALIDITY AND RELIABILITY INTERVIEWS - Controlling for reliability: to have a highly structured interview, with the same format and sequence of words and questions for each respondent. - It is important for each interview to understand the questions in the same way and that the reliability of interviews can be enhanced by: careful piloting of interview schedules, training of interviewers, interrater reliability in the coding of response, and the extended use of closed questions.
(Silverman, 1993).

2. VALIDITY AND RELIABILITY IN QUESTIONNAIRES

* Validity of postal questionnaires can be seen from two viewpoints: (Belson, 1986) - whether respondents who complete questionnaires do so accurately, honesty & correctly. - whether those who fail to return their questionnaires would have given the same distribution of answers as did the returnees * The problem of non-response can be checked and controlled for. It involves follow-up contact with nonrespondents by means of interviewers trained to secure interviews with such people. Then, a comparison is made between the replies of respondents and non-respondents.

2. VALIDITY AND RELIABILITY IN QUESTIONNAIRES

Strategies for maximizing the response rate to the postal questionnaires (to increase reliability):
- including stamped addressed envelopes; - multiple rounds of follow up to request returns; - stressing the importance and the benefits of questionnaires; - stressing the importance of and benefits to, the client group being targeted; - providing interim data from returns to non returners to involve and engage them in the research; - checking addresses and changing them if necessary; - following up questionnaires with a personal telephone call; - tailoring follow-up requests to individual rather than blanket generalized letters; - features of questionnaires itself; - invitations to a follow- up interview; - encouragement to participate by friendly third party; - understand the nature of the sample population in depth.

2. VALIDITY AND RELIABILITY IN QUESTIONNAIRES

* Advantages and disadvantages of questionnaires over interviews:
Advantages - more reliable - more economical Disadvantages - often too low percentage of returns. interviewer: answer questions concerning both the purpose of the interview and any misunderstandings experienced by the interviewee. - if only closed items ->questionnaires may lack coverage and authenticity - if only opened items -> respondents may be unwilling to write their answers for one reason to another. - often filled in hurriedly.

2. VALIDITY AND RELIABILITY IN QUESTIONNAIRES * Sampling considerations: - representative - neither too large nor too small

3. VALIDITY AND RELIABILITY IN EXPERIMENTS

Differences between internal validity and external validity:

Internal validity concerning with the question, do the experimental treatments, in fact, make a difference in the specific experiments under scrutiny?

External validity asking the question, given these demonstrable effects, to what population or settings they can be generalized?

(Campbell and Stanley, Bracht and Glass, and Lewis-Beck)

3. VALIDITY AND RELIABILITY IN EXPERIMENTS

Threats to internal validity
- History - Maturation -Testing - Instrumentation - Selection - Experimental mortality - Selection-maturation interaction

Threats to external validity

- Failure to describe independent variables explicitly. - Lack of representativeness of available and target population. - Hawthorne effect - Inadequate operationalizing of dependent variables. - Sensitization/reactivity to experimental conditions. - Interaction effects of extraneous factors and experimental treatments. - Invalidity or unreliability of instruments. - Ecological validity.

(Campbell and Stanley, Bracht and Glass, and Lewis-Beck)

3. VALIDITY AND RELIABILITY IN EXPERIMENTS

- An experiment can be said internally valid to the extent that within its own confines, its result is credible, but for those results to be useful, they must be generalizable beyond the confines of the particular experiment; in a word, they must be externally valid also.

- A relationship between internal and external validity:

- Without internal validity -> an experiment cannot possibly be externally valid. - An internally valid experiment may or may not have external validity.

4. Validity and reliability in observations :

- Some threats to validity and reliability in observations: + the researcher, in exploring the present, may be unaware of important antecedent events. + informants may be unrepresentative of the sample in the study. + The presence of the observer might bring about different behaviours . + the researcher might go native, becoming too attached to the group to see it sufficiently dispassionately.

4. Validity and reliability in observations :

-The principal ways of overcoming problems of reliability and validity in observations: + Observational techniques are used extensively (both participant and non- participant) to acquire data on real-life settings. (LeCompte and Preissle-1993) + Observations have contextual relevance, both in the immediate setting in which behaviour is observed and in further contexts beyond.(Spindler-1992) + Observation is prolonged and often repetitive. Events and series of events are observed more than once to establish reliability in the observational data. (Spindler-1992)

5. Validity and reliability in tests :

-Wolf (1994) suggests four main factors that might affect reliability: + the range of the group that is being tested. + the groups level of proficiency. + the length of the measure. + the way in which reliability is calculated

5. Validity and reliability in tests :

-Feldt and Brennan (1993) suggest 4 types of threat to
reliability : + individuals(e.g. their motivation, concentration, forgetfulness, carelessness,guessing, their related skills, their usedness to solving the type of problem set,the effects of practice) + situational factors(e.g. the psychological and physical conditions for the test-the context) + test marker factors(e.g. idiosyncrasy and subjectivity ) + instrument variables(e.g. poor domain sampling, errors in sampling tasks,poor question items,scoring errors )

5. Validity and reliability in tests :

- To address reliability there is a need for moderation procedures ( between and after the administration of the test ) to iron out inconsistencies between test markers (Harlen, 1994) including : + statistical reference / scaling tests. + inspection of samples. + group moderation of grades. + post how adjustment of marks. + accreditation of institutions. + visits of verifiers. + agreement panels. + defining marking criteria. + exemplification. + group moderation meetings.

5. Validity and reliability in tests :

-Reliability is also addressed by: + Calculating co-efficients of reliability, split half techniques, parallel/equivalent forms of a test, test, re-test method. + Calculating and controlling the standard error of measurement. + Increasing the sample size. + Increasing the number of observations made and items included in the test. + Ensuring effective domain sampling of items in tests based on item response theory. + Ensuring effective levels of item discriminability and item difficulty.

5. Validity and reliability in tests :

-With regard to validity, it is important to note here that an effective test will ensure adequate : + content validity: is achieved by ensuring that the content of the test fairly samples the class or fields of the situations or subject matter in question and by making professional judgements about the relevance and sampling of the contents of the test to a particular domain. + criterion-related validity: is achieved by comparing the scores on the test with one or more variables(criteria) from other measures or tests that are considered to measure the same factor. + construct validity: is achieved by ensuring that performance on the test is fairly explained by particular appropriate constructs or concepts.

5. Validity and reliability in tests :

+ concurrent validity: can occur simultaneously with another instrument rather than after some time has elapsed + face validity : the test appears at face value to test what it is designed to test. + jury validity : an important element in construct validity, where it is important to agree on the conceptions and operationalization of an unobservable construct + predictive validity : where results on a test accurately predict subsequent performance akin to criterion related validity. + consequential validity : where the inferences that can be made from a test are sound.

5. Validity and reliability in tests :

+ systemic validity: where programme activities both enhance test performance and enhance performance of the construct that is being addressed in the objective.(Fredericksen and Collin ,1989) - To ensure test validity, then the test must demonstrate fitness for purpose as well as addressing the several types of validity mentioned above.

6. VALIDITY AND RELIABILITY IN LIFE HISTORIES - Reliability in life history research hinges upon the identification of sources of bias and the application of techniques to reduce them. Bias arises from the informant, the researcher and the interactional encounter itself. - The validity of any life history lies in its ability to represent the informants subjective reality, that is to say, his or her definition of the situation.

CONCLUSION
The use of reliability and validity is common in quantitative and qualitative, naturalistic research. It is suggested that reliability is a necessary but insufficient condition for validity in research; reliability is a necessary precondition of validity. Researchers have to pay much attention to threats to validity and reliability and the ways to overcome them mentioned above so that these threats can be attenuated