Pusat Pengajian Ilmu Pendidikan

Kolokium Pelajar Pengajian Siswazah PPIP 2005

8.00 - 10.00 pagi. Input Session IV, 16 Disember 2005 (Jumaat)

Quantitative Research Methodology

by
Mokhtar Ismail

Introduction

The purpose of doing research is to add a new knowledge to the existing body of
knowledge in an area of interest. A research problem therefore should not be a tedious
one. It should be carefully thought and polished before going to the field for data
collection. To compose a good research problem is time consuming. Sometimes our first
idea about an issue is not really a research problem as yet, until the idea is discussed and
expanded. The best method of expanding the research problem based on the first idea is
by searching through the journal articles on what other people have done in the area. It is
worth spending more time in polishing up a research problem than going too early to start
the research.

If a research problem needs quantitative approach to address the research questions

related to the problem, the researcher should be ready to equip him/herself with sufficient
knowledge of quantitative research methodology pertinent to the research itself. For that
matter not every body can do research. Only those who are patient, persevere,
hardworking and critical minded can. One should be patient in trying to understand the
statistical concepts that underlie the rationale of making generalization of the research
results. One should be persevere in selecting the most valid instrument for the research.
One should be hardworking in looking for very related models or theories that underlie
the research premise and one should be critical minded in analyzing the research data.

To begin with one should learn about basic descriptive and inferential statistics and types
of research and make sense out of the linkages between the two. Besides, one should also
learn about the sequence of reporting research report: Chapter one to five (or sometimes
more than five) and the skills on how to carry out literature review. The focus of this
paper is on making sense out of the linkages between statistics and types of research
because this constitutes the major difference between quantitative and qualitative
research approach.

Types of Research

Fitting a research problem to a specific type of research is quite a task. Since one has to
be crystal clear on the relationships among the variables in the research problem before
deciding on the types of research to be adopted. There are two major kinds of
relationships, i.e., with or without cause-and-effect. A cause-and-effect relationship
demands experimental, quasi-experimental, ex-post-facto method or probably a time
 2

series design for longitudinal observations. A non cause-and-effect relationship requires a

plain descriptive research describing about the pattern of relationships among the
variables.

Let us take a look at a brief research problem as an example in selecting a type of

research:

Teaching of Science and Mathematics in English creates a lot of problems among

students as well as teachers. The question ‘Do students actually benefited from the
new policy?’ really bothers every citizen.

Based on the above research problem many types of research could be planned depending
on the focus by a researcher. For example, let cause-and-effect relationship be the focus,
i.e., the interest of every citizen would be ‘Did the new policy causes students
performance to drop?’ Thus, the relevant types of research are, a pure experiment, a
quasi-experiment, an ex-post-facto or a time-series design:

Type 1: A pure experiment. This type enables us to manipulate an independent variable

(iv) in order to see the effect on the dependent variable (dv). With regard to the above
research problem, the iv is ‘implementation of new policy’ and the dv is students
performance. Practically, it is not possible to manipulate the iv in the above example,
therefore a true experiment is out of question.

Type 2: A quasi experiment. The only difference between a quasi experiment and a true
experiment is that, in quasi experiment there is no randomization of subjects between
levels of the iv, for instance between control and experimental groups. With regard to the
above research problem a quasi experiment is also out of question for similar reason as
Type 1.

Type 3: Ex-post-facto or causal-comparative. A causal relation could also be established

by causal-comparative method although not as strong as the experimental method. The
method is suitable when iv is already available in the research setting. As for the above
research problem, this type could be the closest in order to search for a cause-and-effect
relationship between the iv and the dv. The iv is already there in the research setting:
‘implementation of new policy’. Our job is to search for the levels of the iv namely
subjects who are being exposed to the new policy and subjects who are not. For example:
if the new policy has been carried out for last three years, the comparison in Mathematics
ability could be made between end of year three pupils this year (who were taught in
English) and beginning of year four pupils (who were taught in Malay).

Type 4: Time series design. A cause-and-effect relationship could also be established

using a time series design. A series of observations based on a defined duration between
observations are recorded for a group of subjects before and after a treatment is given. If
we find that the performance of the subjects are consistently higher after the treatment,
than the effect has taken place and it is caused by the treatment. As for the above research
problem, probably it is not possible to carry out this research design since the treatment
 3

which is ‘the new policy’ has already taken place. Had we known that this policy was
going to be implemented last three years we could have gathered time series data way
before that time, in order to see the effect after the policy is being implemented.

Now let us use the similar research problem and try to fit in the types of research which
are non-cause-and-effect in nature or also known as descriptive research.

Type 5: Survey research. If the focus is not so much on A causes B, but rather the
description of a phenomena such as relationship among variables, survey research is
appropriate. For example, the concern is about views of general public including parents
about the new policy, i.e., how the new policy has affected achievement of their children
in Mathematics.

Type 6: Correlational research. A researcher has prior knowledge about correlations

among variables in previous studies similar to the one in the above research problem. For
instance, there is a high correlation between ability to speak English and number of
English vocabularies of each pupil.

Type 7: R&D type of research. The focus is on development of a prototype and a

validation process to justify its usefulness. With regard to the above research problem,
probably the interest is on developing a new teaching method and this method is going to
be validated by means of an experiment. PTPM students like to do this type of research.
They develop courseware and they validate its effectiveness by means of an experiment.
Psychometric students too carry out this type research. They develop tests and validate
them by presenting validity evidences of various kinds: content, predictive and construct.

Type 8: Evaluation research. The focus is on evaluating an event by means of the above
types of research and to make judgment about its usefulness. This type of research is
probably not truly quantitative due to the elements of value judgment made by the
researcher. With regard to the above research problem, the focus is on for example the
effectiveness of teaching of Mathematics courseware currently used by teachers. There is
a slight difference between Evaluation Research and Program Evaluation. The latter is for
decision making purposes rather than adding new knowledge to the body of existing
knowledge in the area of interest and is usually based on existing program evaluation
models pioneered by Ralph Tyler.

There are sometimes overlaps in terms of procedures between each research type: 1
through 8, and sometimes it is not easy to categorize them objectively according to each
of the type. Nevertheless a researcher should have some idea on what is the best name to
be assigned to his or her research type.

If the above types do not fit a researcher’s focus, probably qualitative research approach
is the answer. This approach requires totally different philosophy in terms of ‘social
reality’. As for the above research problem the interest is on the process rather than the
outcome. For example, the researcher is interested in the mechanics of pupil
comprehension process when they listen in English and think in their native language.
 4

How does learning take place under this condition? Is there hindrance along the learning
process? These are issues most appropriately addressed by a qualitative research
approach.

Hypothesis Testing and Types of Statistical Analysis

Generalization process from sample to population is the intention of a quantitative as

opposed to a qualitative researcher. In research, only one sample of subjects is studied
and based upon characteristics of that sample, generalization is made back to the
population where the sample is formerly chosen. Normally, characteristic of the sample is
represented by the sample mean. In quantitative research the big question is whether that
sample mean represents the population mean? The question is quite tricky and should be
clearly understood by every quantitative researcher.

Data collection activity normally starts with a sampling procedure when a random sample
is carefully chosen. The sampling has to be random because we have to make use of the
rationale of the probability theory that underlies the distribution of the sample means
since in research we only pick one sample and compute the mean. This mean is one the
many means in the distribution of the sample means if we endlessly pick up a sample and
compute the mean from a defined population. The distribution of the sample means,
picked up endlessly, will actually form a normal distribution. This is known as the
Central Limit Theorem. In addition, the standard deviation of the distribution of the
sample means is known as standard error, which can be estimated by the standard
deviation of the sample and the sample size, and the mean of the distribution of the
sample means is of cause the population mean.

The value of standard error of the mean has bearing on the representativeness of a
sample mean in a population. If it is small the implication is that the sample means are
close to population mean and vice versa. In research we expect the standard error to be
small because what it means is that, the sample mean that we obtain from our study is
close to the population mean. In other words, our sample is representative.

The question is how close is the sample mean to the population mean in order for a
researcher to decide that his or her sample mean represents the population mean? To
decide on the closeness, statisticians have developed hypothesis testing procedure. To
make the testing procedure meaningful they start with the concept of null hypothesis or
empty hypothesis. What they want to say is that, if the sample mean is truly close to or
far from the population mean, that should happen beyond reasonable doubt (we borrow a
legal term). It statistics, we say it differently, beyond purely chances alone.

The null hypothesis is always presented in the following form: There is no significant
difference between A and B. In our above example, we could say that, there is no
significant difference between sample mean and population mean (A and B can also be
correlations or variances). If we can reject the null hypothesis beyond reasonable doubt,
we are sure that our sample mean is close to or far from population mean, therefore our
 5

sample mean represents the population mean. We actually do not know what really
happens in the population. That is why we do research.

The next question is how to measure the distance of the sample mean to the population
mean so that we are able to reject the null hypothesis beyond reasonable doubt? To start
with, we should learn about the concept of z-score, i.e. a kind of derived score or
standardized score transformed from a raw score. In any frequency distribution of scores,
we can always transform raw scores into derived scores. One of them is z-score. A z-
score is defined as the difference of a raw score against its mean in standard deviation
unit (a z-score is raw score minus mean divide by standard deviation). This is just a linear
transformation between raw scores into the standardized scores. The frequency
distribution of standardized score is not necessarily normal, but if it is normal it is known
as a standard normal frequency distribution or we usually call it as normal distribution.
The normal distribution has the well known characteristics namely: 68 percent of the
cases (we call it cases because y axis represents frequency of the distribution, so the area
under the curve actually represents the density of frequencies or cases) are between 1
standard deviation below and above the mean, 95 percent of the cases are between 2
standard deviation below and above the mean, and 99 percent of the cases are between 3
standard deviation below and above the mean.

Let us go back to the original question of how to measure and later decide on the
closeness between a sample mean to the population mean by applying the null hypothesis
concept. After getting the sample, we compute sample mean and sample standard
deviation. From the sample standard deviation and sample size, we compute standard
error of the mean. The standard error tells us about the standard deviation of the
frequency distribution of sample means, if we pick up samples endlessly and compute
their means respectively (but remember in research we only randomly pick up one
sample, and yet we can still estimate the standard error). The next job is to compute the
difference between the sample mean and the population mean (or hypothesized mean) in
terms of standard deviation unit using the z-score concept where the numerator is the
difference between sample and population mean and denominator is standard error. This
activity is known as computation of test statistic. In order to know whether we can reject
the null hypothesis, we have to compare the test statistic value with the critical value. As
for our example, we have to compare the z-value that we get with the z-value in the
standard normal frequency distribution which has the characteristics as mentioned above.
If the test statistics value falls in the critical region, i.e. more than two standard deviations
below or above the hypothesized mean, normally the decision is to reject the null
hypothesis. The rationale is: if the null hypothesis is true, it is highly unlikely that the test
statistics value to appear (the probability is less than five percent), but it does appear, so
the decision is to reject the mull hypothesis.

Practically, the z-score concept in computing the test statistic and testing it, is applicable
to other types of test statistic. The types of major test statistic are given on Appendix A.
The most common test statistic is independent sample t test for testing difference of two
means. t test is actually the extension of z-score concept, and the t test is meant for
smaller samples as compared to z-test.
 6

Conclusion

The intention of this paper is to highlight the most crucial statistical aspect of quantitative
research methodology in relation to the types of research. There are a lot more other
aspects that have to be considered which are not discussed.

Copyright by Mokhtar Ismail 2005

 7

Appendix A

TYPES OF STATISTICAL ANALYSIS

Number of dependent variable (dv)

________________________________________
one two or more
(univariate) (bivariate no iv or dv) (multivariate)

___________________________ ______________________
between subject within subject relationship: between subject:
cannonical, manova,
_____________ _____________ discriminant mancova
one iv two or more iv two levels two or
more
____________ levels
two more than
groups two groups Correlated
t-test

Repeated
Measue
two or more way ANOVA
ANOVA

Oneway
ANOVA

Independent
Sample t-test
bivariate correlation,
multiple regression

Notes

Correlations: statistical and practical significance

Counterparts: parametric and non-parametric
Input layout for ANOVA: one or more way
 8

ANCOVA: relationship between covariate and dv

Analysis: choose the most efficient
Pls check with sugino notes on test statistics.

Pls check with sugino notes on test statistics.

Identification of variables
Example on quantitative data analysis
Validity of an experiment (threats)
Research Plan
SPSS package
Choice of research approach: quantitative versus qualitative
Sampling
Summary