Operationalizing research

Broadly speaking, research comprises two phases: the thinking phase and the action phase.

Thinking Phase:

identifying a problem,

formulating a research question and hypothesis

identifying study variables.

This step is important because it provides direction to the action phase of research, which mainly deals
with the study methodology, analyses, and dissemination of findings.

It is circular in nature rather than linear

There is a lot of back-and-forth movement involved between the different activities in both the
phases. In the thinking phase, researchers continually evaluate ideas, review literature, rethink, and
reconceptualize problems before finalizing the study problem tatement.

The action phase, study results help to create a new knowledge base or generate new ideas for the next
study.

Operationalizing the research purpose

To describe how to formulate and articulate the thinking phase of research.

Steps in operationalization:

1) Research problem, 2) Research question 3) Hypothesis 4) Operational definition.

1) Research problems:
A problem is defined as an interrogative statement that questions the interrelationship of two or more
variables.

well-written research problem statement should be able to answer the following three questions:

1-What is being studied?

To identify a research topic that is to be studied. A researcher starts with a general topic of interest or
problem to be solved and continuously refines it to have a more focused topic.

Example: studying influenza immunization rates.

2-Who is being studied? Segmentation and target sample

Identifies the population in which the study would be conducted. The goal is to characterize the
population in terms of social, demographic, and economic characteristics. This helps a researcher focus
on issues that are prevalent in a given population segment or community.

Example: “studying influenza immunization rates among inner-city poor populations.”

3-Why is this being studied?

Emphasizes the relevance or significance of the study and its contribution to the knowledge base.

“Why are immunization rates among inner-city poor populations the lowest in the country?”

1
Characteristics of a good problem

first, a problem should express a relationship between two or more variables

second, the problem should be well defined, clear, and free of any ambiguity;

third, the problem must be ready for empirical investigation, This means that the relationship between
variables should be stated and the variables should be measurable

Most Common sources of research problems

1 improving clinical practice

2 exploring a pattern of incidents

3 testing folk wisdom

4 understanding phenomena from the insider’s perspective

5 tackling current issues

6 inconsistencies in the literature

7 testing a theory

8 testing practice theories

9 exploring variations in a dependent variable

10 providing an evaluation

11 implementing and studying action

12 replicating a study.

2) Research questions
Research questions are interrogative statements and, specifically, indicate what researchers want to find
out about a problem area.

Research questions focus on describing variable(s), examining relationships among variables, or

determining differences between two or more groups on a single or multiple variable(s) of interest.

Some examples of research questions include:

1 What is the incidence of influenza in white and nonwhite people?

2 What are the differences in health beliefs about influenza vaccination immunization between white and
non-white people?

Characteristics of a well-written research question

In contrast to research problems, research questions are very specific and precise.

Research questions should be well written and clinically relevant, and have the potential to yield
information to improve pharmaceutical practice and policy.

The characteristics of a well-written research question can be summarized by the mnemonic FINER which
stands for feasible, interesting, novel, ethical, and relevant

(Feasibility factors include: enrolment, multidisciplinary approach, technical expertise, time

requirements, and funding)
2
There are no standard guidelines for writing questions.

There are two components to any research question:

the stem of the question, which directs the research process.

the topic of interest, which is simply what the question is about or the focus of the study.

For example, in the question “What are the characteristics of successful practitioners?” the stem is “what
are” and the topic of interest is “characteristics of successful practitioners.” Based on the number of stems
and topics, a research question can be classified as a simple or complex question.

Research generally addresses three basic types of research questions:

Descriptive; prevalence and incidence studies

Relational; relationship between different phenomena. For example, the proportion of men willing to
quit smoking compared with women in the last year is essentially studying the relationship between
gender and willingness to quit smoking.

Causal; Example does an intervention program using a motivational interviewing technique improve
quitting in a smoking population? The Gold Standard is RCT.

3( Hypothesis
A hypothesis is a statement about the expected relationships between two or more variables. Hypotheses
serve as bridges between theory and the empirical study.

Other words,

hypothesis is a specific statement in a testable form, that consists of at least one specific relationship
between two variables (or concepts), which are measurable.

For example, “Patients with a greater knowledge of asthma have a significantly higher rate of adherence
to the medication regimen.”

A) Simple versus complex hypotheses

one independent variable and one dependent variable

Hypothesis Independent Dependent

“Patients with greater knowledge of asthma have a amount of adherence.
significantly higher rate of adherence to the medication knowledge (greater
regimen than patients who have little knowledge about their versus lesser)
disease.”
Patients with greater knowledge of asthma and better access amount of adherence,
to care have a significantly higher rate of adherence to the knowledge (greater incidence
medication regimen, decreased hospitalization rate, and versus lesser) of
incur lower costs of care than patients who have little AND access to care hospitalization,
knowledge about their disease or have a barrier to access to (some versus none) and costs of
care care.

B) Nondirectional versus directional hypotheses

In a nondirectional hypothesis, a relationship between variables is predicted but the researcher is unsure
or not confident about the direction of the outcome.

A directional hypothesis is stated as “When X increases, Y increases” (positive or direct relationship) or

“When X increases, Y decreases” (negative or indirect relationship).

3
C) Associative versus causal hypotheses

Association can be either positive or negative, is similar to the direction hypothesis.

Causal hypotheses are always directional and independent variables must precede dependent variables.

For example, a causal hypothesis can be stated as “A new treatment regimen will cause a difference in
depression.”

D) Statistical versus Non-statistical

Statistical hypotheses are usually used to test the relationship statistically and determine whether the
observed relationship is due to chance or is a true relationship

4(Operationalization definitions
Moving from 1-3, same definition of a variable can be used at different times and in different settings.

4
 Measurement related issues
Measurement forms the basis for empirical health services research including pharmaceutical
practice and policy research.
In healthcare, critical policy and practice discussions are based on cost, quality, and access. These
concepts or constructs have to be measured and analysed using variables, to evaluate the
performance of a healthcare system at the patient, family, institution, and population levels.
Measurement and further analysis are dependent on the properties of the variable. The levels of
measurement can be classified as nominal, ordinal, interval, and ratio, based on the properties
of the quantitative and/or qualitative variables.
The type of measurement of variables determines the statistical analysis to be used. This holds
true for both dependent and independent variables.

Reliability & Validity of Measurements

A) Reliability
Reliability, as discussed earlier, ensures that the measurement is consistent.
The reliability assessments can be grouped into two major classes:
Measurement of stability: evaluate relationship or correlation of measures across time or
evaluators, and examples include the test–re-test method and interrater reliability.
Measurement of equivalence: evaluate relationship or correlation between two sets of
instruments, and examples include split-half, parallel form, and internal consistency methods.

Test–re-test method: analyses measures obtained from the same participants across time using
the same instrument. This method is often used to ascertain the reliability of physiological
measurements such as blood glucose and blood pressure.
The correlation between the two measures on the same participants provides the correlation
coefficient.
5
 Two variables that are continuous (e.g., weight, cholesterol)
and are paired (i.e., each case has two values - one for each
Pearson's correlation variable).
two continuous variables OR two ordinal variables
Spearman's correlation OR one ordinal and one continuous variable
chi-square test of » Two variables that are nominal/dichotomous (e.g.,
independence males/females).
» There are two or more groups in each variable.
Loglinear analysis Two or more variables that are categorical (e.g., gender,
employment status, etc.).

Disadvantages of the test–re-test method include inconvenience and reactivity.

Due to multiple measurement processes it is inconvenient to participants and researchers.
Reactivity refers to change in the underlying construct due to testing. For example, patients
responding to a question related to compliance are sensitized to the issue of compliance and
thus provide responses that might reflect an improvement in compliance.

6
Interrater reliability involves analysis of measures obtained from the same participants by
different evaluators using the same instrument. For example, pharmacists’ counselling time can
be measured by two independent observers. Simply it can be measured by percent agreement.
However, coefficient is statistically factor used for interrater reliability.
Cronbach’s α when the judgments are quantitative
Cohen’s k coefficient (Cohen's kappa coefficient (κ)) is used for reliability involving two nominal
measures.
Fleiss’s k is used for measures involving multiple nominal measures.

Internal consistency, which is the consistency of people’s responses across the items on a
multiple-item measure.
Cronbach’s a is used to calculate the internal consistency of measures based on continuous
measures.
Kuder–Richardson coefficient (KR20 or KR21) is used to calculate reliability for nominal
measures.
Split-half correlation. This involves splitting the items into two sets, such as the first and second
halves of the items or the even- and odd-numbered items. Then a score is computed for each set
of items, and the relationship between the two sets of scores is examined.
The parallel-form method is an extension of the split-half method in which two parallel
questionnaires are administered to each participant consecutively.
In Split half and parallel form, the two halves should be designed to measure the same underlying
construct.

B) Validity
Validity, ensures that the instrument developed for measurement purposes truly represents the
underlying construct.
Validity simply means that a test or instrument is accurately measuring what it’s supposed to.
For example, a test of intelligence should measure intelligence and not something else (such as
memory).
Internal and External Validity
A distinction can be made between internal and external validity. These types of validity are
relevant to evaluating the validity of a research study / procedure.
Internal validity refers to whether the effects observed in a study are due to the manipulation of
the independent variable and not some other factor. In-other-words there is a causal relationship
between the independent and dependent variable.
Internal validity can be improved by controlling extraneous variables, using standardized
instructions, counter balancing, and eliminating demand characteristics and investigator effects.

7
External validity refers to the extent to which the results of a study can be generalized to other
settings (ecological validity), other people (population validity) and over time (historical validity).
External validity can be improved by setting experiments in a more natural setting and
using random sampling to select participants.
Some specific examples of threats to external validity:
Is your sample selected randomly? If not, it may be open to selection bias.
Have you included a pre-test? In some experiments, pre-tests may influence the outcome. A
pre-test might clue the subjects in about the ways they are expected to answer or behave.
Are your participants taking multiple versions of the same test? If so, the practice effect might
influence your results. For example, the Wechsler Intelligence Scale for Children is highly
influenced by the practice effect.
Is your sample composed of a homogeneous population, like all low achievers or all high
achievers? If so, your results probably won’t be generalizable to the “average” person.
Are the results of your study tainted by the Hawthorne effect? Your study participants may be
behaving differently because they know they are in an experimental study.
Replicability
it refers to whether the results from your test or experiment can be replicated if repeated exactly
the same way in other experiment by another researcher. To demonstrate replicability, you must
provide statistical evidence that shows your results can be used to predict outcomes in other
experiments. An example of evidence that can suggest replicability is the probability value, or p-
value. If your test produces a small p-value (usually less than .05 or 5 percent), it suggests your
results are not due to chance. If your results can be replicated by different researchers in
different experiments, then your results are called “robust.”
Remember that Replicability is different than that Verifiability & Reliability

Assessing the validity of test

There are two main categories of validity used to assess the validity of test (i.e. questionnaire,
interview, IQ test etc.): Content and Criterion validity.

8
 - Content related validity
1) Face Validity
This is the least sophisticated measure of validity. Face validity is simply whether the test appears
(at face value) to measure what it claims to.
Tests wherein the purpose is clear, even to naïve respondents, are said to have high face validity.
Accordingly, tests wherein the purpose is unclear have low face validity.
A direct measurement of face validity is obtained by asking people to rate the validity of a test
as it appears to them. This rater could use a likert scale to assess face validity. For example:
- the test is extremely suitable for a given purpose
- the test is very suitable for that purpose;
- the test is adequate
- the test is inadequate
- the test is irrelevant and therefore unsuitable

It is important to select suitable people to rate a test (e.g. questionnaire, interview, IQ test etc.).
For example:
1st: Individuals who actually take the test would be well placed to judge its face validity. Also
2nd: people who work with the test could offer their opinion (e.g. employers, university
administrators, employers, HCPs).
Finally, the researcher could use members of the general public with an interest in the test (e.g.
parents of testees, politicians, teachers etc.).
The face validity of a test can be considered a robust construct only if a reasonable level of
agreement exists among raters.

9
It should be noted that the term face validity should be avoided when the rating is done by
"expert" as content validity is more appropriate. Yet, expert can provide judgment of face
validity.
Note:
Having face validity does not mean that a test really measures what the researcher intends to
measure, but only in the judgment of raters that it appears to do so. Consequently it is a crude
and basic measure of validity.
A test item such as 'I have recently thought of killing myself' has obvious face validity as an item
measuring suicidal cognitions, and may be useful when measuring symptoms of depression.
It is possible for a test item to lack face validity but still have general validity and measure what
it claims to measure. This is good because it reduces demand characteristics and makes it harder
for respondents to manipulate their answers.
2) Construct validity (here the discussion is for the sub type: Content validity)
Content validity, sometimes called logical or rational validity, is the estimate of how much a
measure represents every single element of a construct.
It refers to the representative nature of the items to capture the underlying dimension of the
construct. It presents the relationship of items to the dimensions of the underlying construct.
For example,
measurement of healthcare quality, requires items or questions related to:
the structure of the healthcare system, process of obtaining healthcare, and outcomes of
healthcare obtained.
In addition to expert opinions, Analytical measures such as content validation ratio, content
validation form, and content validation index can strengthen the content validity.
Note:
Content validity is also similar to face validity. However, they both use different approaches to
check for validity. Face validity is an informal way to check for validity; anyone could take a test
at its “face value” and say it looks good. Content validity uses a more formal, statistics-based
approach, usually with experts in the field. These experts judge the questions on how well they
cover the material
Examples of measurements that are content valid:
Height (construct) measured in centimeters (measurement).
Physics knowledge (construct) measured by the AP exam (measurement).

2. A) Convergent validity tests that constructs that are expected to be related are, in fact,
related.
2. B) Discriminant validity tests that constructs that should have no relationship do, in fact, not
have any relationship. (also referred to as divergent validity)

10
For example, instruments to measure pain and overall quality of life will be strongly associated
because they measure similar concepts. Conversely, pain measures are less likely to be
associated with perceptions on economy as these are dissimilar concepts.
Let’s say you were researching depression in college students. In order to measure depression
(the construct), you use two measurements: a survey and participant observation. If the scores
from your two measurements are close enough (i.e. they converge), this demonstrates that they
are measuring the same construct. If they don’t converge, this could indicate they are measuring
different constructs (for example, anger and depression or self-worth and depression).

- Criterion related validity

attesting to the relationships between the instrument with other measures, based on theory and
practice.
For example
A job applicant takes a performance test during the interview process. If this test accurately
predicts how well the employee will perform on the job, the test is said to have criterion
validity.
A graduate student takes the GRE. The GRE has been shown as an effective tool (i.e. it has
criterion validity) for predicting how well a student will perform in graduate studies.
The first measure (in the above examples, the job performance test and the GRE) is sometimes
called the predictor variable or the estimator.
The second measure is called the criterion variable as long as the measure is known to be a valid
tool for predicting outcomes.
Criterion validity includes: concurrent, predictive and postdictive.
1) Concurrent validity measures the test against a benchmark test ( at the same time) and high
correlation indicates that the test has strong criterion validity. For example: new IQ or
personality test might be compared with an older but similar test known to have good validity
already. OR compliance measured using self-reports can be validated by comparing responses
with the drug levels in blood or urine measured at the same time.
Advantages:
It is a fast way to validate your data.
It is a highly appropriate way to validate personal attributes (i.e. depression, IQ, strengths and
weaknesses).
Disadvantages:
It is less effective than predictive validity to predict future performance or potential, like job
performance or ability to succeed in college.

11
2) Predictive validity is a measure of how well a test predicts abilities. It involves testing a group
of subjects for a certain construct and then comparing them with results obtained at some point
in the future. For example, a prediction may be made on the basis of a new intelligence test, that
high scorers at age 12 will be more likely to obtain university degrees several years later. Or the
prediction results from GRE exams
Weakness pf predictive validity:
In the university selection example, this approach does not test the students who failed to attend
university, due to low grades, personal preference or financial concerns. This leaves a hole in the
data, and the predictive validity relies upon this incomplete data set, so the researchers must
always make some assumptions.

3) Postdictive validity: if the test is a valid measure of something that happened before. For
example, does a test for adult memories of childhood events work?

12