Quantitative

Research Designs

Ross Donohue
Types of Quantitative Designs

 Experimental Designs
– True experiments
– Quasi-experiments

 Non-experimental Designs
– Correlational Field Studies (surveys)
Quasi-Experimental Designs
 One Group Pretest-Posttest Design

NR E: O1 X O2

 Advantage: provides a base line

 Disadvantage: no control group threats to
internal validity
 Can use nonequivalent dependent variable
design - additional theoretically unrelated
DV’s to establish discriminant validity
 Quasi-Experimental Designs
 Pretest-Posttest Nonequivalent Comparison
Group Design

NR E: O1 X O2
NR C: O1 ~X O2

 Weak – participants self-select

 Moderate – intact groups formed on the basis
of unrelated factors
 Strong – intact groups, but researcher
assigns treatment conditions
 Design can also be improved by switching
replications
Quasi-Experimental Designs
 Interrupted Time Series With a Non-equivalent No-
treatment Comparison Group
NR E: O1 O2 O3 X O4 O5 O6
NR C: O1 O2 O3 ~X O4 O5 O6

 If the treatment has had an effect, the slope or level

of pre-test observations will differ after the treatment
 Can check for any effects due to regression towards
the mean
 Comparison group allows researcher to control for
history effects
Nature of Surveys
 Inquiry into a social or human problem
 Aim: Establish relationships between IVs &
DV
 Variables measured by multi-item
questionnaire scales, or by single item
variables
 Field study
 No manipulations
 Correlational basis
 Multivariate analysis usually
When to Use a Survey/Correlational
Field Study
 Test a theory of alternative predictions
 Test RQs & hypotheses on a large
number of people
 Real-life setting
 Generalise the findings
 Solid literature base
 Access several IVs and controls
Advantages of Surveys
 Speed
 Breadth
 Standardisation
 Generalisability
 Flexibility in variable selection
 Cost - efficiency
 Able to be analysed statistically
 Computer Scoring
 Unobtrusiveness
Problems With Surveys
 Standardisation loss in data “richness”
 Can’t deal with the context of social life
 Design inflexibility
 Unreliable measures
 Low power
 Inadequate sampling designs
 Meaningless responses
 Non-pretested measures
 Results generalised beyond the sample
 Cross-sectional data
 Method problems
Characteristics of Interpretable, Robust
Correlational Field Studies
 Variables to be measured are chosen
based on a strong theoretical basis
 Measurement of dependent and
independent variables
 Measurement of control variables
 Measurement of multiple independent
variables
Characteristics of Interpretable, Robust
Correlational Field Studies
 Inclusion of mediator or moderator
variables where theoretically needed
 Longitudinal designs used rather than
cross-sectional designs
 Valid and reliable measures are used
– Validity (construct, criterion, content)
– Reliability (internal consistency, test-retest)
Characteristics of Interpretable, Robust
Correlational Field Studies
 Sample chosen to answer the question
– Probability Sampling
• Simple random sampling
• Systematic random sampling
• Stratified random sampling
– Nonprobability Sampling
• Convenience sampling
• Purposive sampling
• Quota sampling
• Snowball sampling
 Sample size provides adequate power
Characteristics of Interpretable, Robust
Correlational Field Studies
 Valid types of data are gathered
– Subjective/objective/hard data
– Same source vs different source
– Self-report vs others’-report
– Individual vs Pair vs Group vs
Organisational Level
Characteristics of Interpretable, Robust
Correlation Field Studies
 Common method variance is reduced
– Longitudinal data
– Subjective and objective data
– Measures of the predictor and criterion variables
from different sources
– Counterbalancing of the question order
– Harman’s single-factor test
– Partial correlation or confirmatory factor analysis
procedures
– Procedures that protect respondent anonymity and
reduce evaluation apprehension
Mail-Out & Internet Surveys
 Advantages
– Geographic flexibility
– Low cost
– Respondent convenience
– Researcher absence – anonymity
 Disadvantages
– Researcher absence – confusion
– Selection bias
– Poor response rate – major problem
Expected Responses Rates
 Baruch (1999) average response rate
from175 published management studies –
55.6% (SD=19.7); lower with top
management respondents – 36.1%
(SD=13.3)
 Harzing (1997) International mail surveys
– Japan (28.6%), Europe (22.9%), US (11.4%),
Hong Kong (7.1%)
 Cycyota & Harrison (2006) – average
response rate of 34% (SD=17) with executive
respondents
How to Increase Response Rates from
Mail-Out Surveys
 In order of Importance:
1. Follow-up
2. Financial incentives
3. Prior notice
4. Special postage
5. Sponsor
6. Stamped return
7. Personalisation
8. Interest in survey topic
9. Questionnaire length