Experimental Methodology

Maggie De Young Julie Harris Heather McCullough Ryn Nasser Wayne Stone

Basic Terminology
Experiment Experimental groups
Control group Experimental Subject

Treatment Control

Basic Terminology
Field experiments Laboratory experiments Pretest/posttest Randomization Internal validity Reproducibility

Four Cell Experimental Design

Value of DV prior to treatment Control Group Experimental Group Value of DV after treatment

Other design issues/concepts

Random number tables used to ensure randomization of subject assignment to groups Factorial designs experimental designs that involve two or more independent variables

Evaluating the Experiment

Null hypothesis a statement that no significant difference exists between the control and experimental groups. The evaluation is conducted using ttests, chi-square tests, and analysis of variances (ANOVA) tests.

Sensitivity
Sensitivity experiments ability to detect relatively small effects. Increase sensitivity by increasing the number of subjects (this decreases the chance for random/experimental error)

External Validity
External validity the extent to which a sample is representative of the target population:

Can the results be generalized? If so, what is the population to which it can be generalized? Can the results be generalized to the real world?

Easier to justify generalizations for field experiments than laboratory experiments because lab experiments are so artificial

Ex Post Facto Study

Ex post facto = Latin for after the fact. Example of a quasi-experimental design Quasi-experimental design = Not a true experimental design because it lacks control, but it seeks to analyze an event that has already happened in an effort to isolate the causal relationships of the event

Considerations with experiments

Most appropriate technique to test hypotheses which involve causal relationships because of the control mechanisms built into the method. Ethical considerations may preclude the use of experiments on human subjects. Biases have to be carefully addressed with this methodology. Costs of control and randomization are a consideration.

An Example of the Experimental Methodology:

Wildemuth, Barbara M., Charles P. Friedman, and Stephen M. Downs. Hypertext versus Boolean Access to Biomedical Information: A Comparison of Effectiveness, Efficiency, and User Preferences. ACM Transactions on Computer-Human Interaction. 5.2 (1998): 156-183.

Research Questions
To what extent does information retrieved from a database help students solve problems in bacteriology? Does this effect vary as students levels of personal domain knowledge increase? Is Boolean or hypertext access to a database more effective in supporting problem solving? Is one of these database access mechanisms more efficient than the other, in terms of its effect on problem-solving performance? Do students prefer one access mechanism over the other?

Experimental Method and Our Article

Randomization: Simple**
Sample size: 82 Control exerted through independent variables: 1) access mode to which student is assigned 2) problem set order 3) assessment occasion Hypothesis: (1) database use is an effective supplement to personal knowledge in a problemsolving context and (2) the level of the searchers personal knowledge in the relevant domain will affect the number and variety of search terms selected

Methodology and Our Article

Randomization? Pretest and Posttest Statistical tests
ANOVA Chi-square T-test

Pretest/Posttest Procedure: 1994 & 1995 1st year Medical Students

Pretest given in November before class in microbiology.
Without database = own knowledge With database

Posttest given in March after class in microbiology

Without database = own knowledge With database

Mean test scores

Without Database
November 1994 13 1995 10 March 55 48

With Database
November 1994 65 1995 60 March 89 80

ANOVA Testing of Assessment (mean scores)

df F 1994 (n-40) 1,36 22.18 1995 (n=42) 1,38 33.4

p 0.0001 0.0001

Total Improvement Mean Scores

November March Boolean Hypertext Boolean Hypertext Mean Mean Mean Mean 1994 (n=40) 48.7 57.1 35.2 32.9 1995 (n=40) 50.3 49.4 30.9 32.8

1995 Students
Boolean (n=22) Hypertext (n=20) Mean Mean November* March**
0.572 0.645 0.563 0.548

*November test time averaged 96 minutes, **March test time averaged 50 minutes.

Chi-square Testing
The researchers used chi-square testing to determine if the students preferred hypertext or Boolean access. The researchers conclude there was no significant difference in effectiveness or efficiency between hypertext and Boolean. Chi-square results indicated students preferred Boolean access over hypertext.

Second Look Effect

Students can take extra time to look at questions. Jog memory. Operates equally in both groups. Should not constitute any additional bias.

Discussion of Results
The interaction between system preference and system assigned did not have a statistically significant effect on performance. Students with high or low personal knowledge can retrieve relevant information from a database. Both access methods were deemed equally helpful and used with equal efficiency. Hypertext might encourage persistence; Boolean user might conclude prematurely that no useful information exists.