Running head: A CRITICAL ANALYSIS OF CHENS RESEARCH 1

A Critical Analysis of

Baiyun Chens Research

Effects of Advance Organizers on Learning and

Retention from a Fully Web-Based Class

Younga Machorro

California State University Monterey Bay

A CRITICAL ANALYSIS OF CHENS RESEARCH 2

A Critical Analysis of Baiyun Chens Research

Effects of Advance Organizers on Learning and Retention from a Fully Web-Based Class

This paper is a critical analysis of Baiyun Chens research Effects of Advance

Organizers on Learning and Retention from a Fully Web-Based Class written in 2007. The

research will be reviewed in four areas: introduction, research procedures, research results, and

discussion of results. After that, the value of the research to the academic community and

recommendations for further research or application will be discussed.

Introduction

This study is an experimental research. There were two treatment groups and a control

group, and the treatment variable of advance organizer was controlled by the researcher. Both

quantitative and qualitative methods were used to investigate the effects of advance organizers

on learning and retention. However, quantitative method was mainly used and qualitative method

such as a survey and interviews were conducted to supplement the quantitative results with

contextual information (Chen, 2007, p. iii). Most of the discussion was based on the result of

the quantitative research method as well.

The stated purpose of the study is to investigate the short-term and long-term effects of

two kinds of advance organizers (AOs), a visual concept map and a text outline (Chen, 2007, p.

iii). The problem identified in the study was a paucity of research-based evidence in using

advance organizers in fully Web-based courses (Chen, 2007, p. 7), although the use of

multimedia and the Internet in a fully Web-based learning environment brings challenges of

cognitive overload and learner disorientation (Dias & Sousa, 1997, as cited in Chen, 2007, p. 5).

The cognitive theories including Ausubels assimilation theory and Mayers assimilation

encoding theory are being studied in this research.

A CRITICAL ANALYSIS OF CHENS RESEARCH 3

Two null hypotheses are explicitly and clearly stated: (1) there is no difference in the

short-term knowledge-based and performance-based learning achievements among students in

the concept map, outline and control groups (Chen, 2007, p. 7), and (2) there is no difference

in the long-term knowledge-based and performance-based learning achievements among

students in the concept map, outline and control groups (Chen, 2007, p. 8).

The researcher expresses a positive bias in describing the subject of the study. As one of

the explanations attributing the negative results of the research, the researcher points out the

characteristics of the student population possessing high learning ability based on their

demographic information and interview data (Chen, 2007, p. 91). However, this judgment

about the subject population not based on any proper pretest result seems arbitrary hence lacking

credibility. Moreover, it nullifies the research finding identified by the researcher about the

positive effect of AOs on low-ability learners.

The researcher makes a convincing case that the research hypotheses are important to the

study through the identification of the problem and solid literature review. She points out there

was insufficient research that evidences the effectiveness of AO in technology-enhanced

environments, and no research in fully online environments. She emphasizes the need for

empirical evidence for using organizers as one of the best practices in Web-based courses

(Chen, 2007, p. 14).

Research Procedures

This study is a posttest-only randomized experimental research. It was conducted in a

fully online environment; both the instruction and tests were conducted at a time and a place of

students choice during the instruction week without any instructor present. The participants were

112 college students who were enrolled in a Web-based course. The researcher obtained their
A CRITICAL ANALYSIS OF CHENS RESEARCH 4

written agreement to participate in the study before the procedure started. The participants were

randomly divided into three groups: two treatment groups and one control group. The treatment

of AO was given for module 2 during the second week of the course. The first treatment group

was given access to a multimedia concept map and the second treatment group was given access

to a textual outline while the control group was not given any advance organizer. There were two

sets of posttests, one at the end of the one-week long module 2 instruction to test the short-term

effect of AO and the other four weeks after that to test the long-term effect. Both sets of posttests

consist of a quiz of nine multiple choice questions and a test of three essay questions based on a

scenario. The researcher explains the multiple-choice question quiz is to measure knowledge

acquisition and the scenario test is to measure knowledge application. Besides two sets of

posttests, an online survey and interviews via phone or web chat were conducted as well. The

survey was open online during the treatment week and all study participants responded. Six

students during the treatment week and nine students four weeks after that were randomly

selected and interviewed.

This study is sufficiently large and randomized, so it seems safe to state the distribution

of students with various prognostic factors is balanced. According to Urschel et al. (2001), if the

number of each group is less than 30, randomized trials run the risk of comparing (p. 98).

However, in this study, each group had more than 30 students at any point of the research period.

Additionally, the sampling procedures were randomized using Generate groups feature inside

WebCT (Chen, 2007, p. 51).

Despite the sufficiently large group size and randomized sampling procedure, whether the

sampling procedures produced a sample that is representative of the target population is

questionable. As the researcher admits in the limitation section of the study, the generalizability
A CRITICAL ANALYSIS OF CHENS RESEARCH 5

of this study is only suitable for four-year college students in fully Web-based courses in the

subject of Health Care Ethics. The population is disproportionately distributed across gender and

ethnicity, since majority of the participants in this study are white, female, and Caucasians (p.

62) However, this study seems to target online learners in general judging from the researchers

recommendations for practice based on the study.

The measurements also could have been improved to be sufficiently valid and reliable.

Above all, the number of questions of the posttests was too small. As the researcher admits, with

such small numbers of test items and subjects, if one student did not take the testing seriously,

that would affect the test results greatly and distort the research result. That would have been

very possible considering students knew that the test results were not going to affect their final

grade and this was a fully web-based course where even the tests were taken with no supervision.

The content of the tests, especially the knowledge tests, however, reflects the course content

directly and raises the possibility of consulting the textbook, classmates, or other resources. In

fact, during the interview of the study, all 10 interviewees admitted to using either textbook or

lecture notes to some degree during the online quizzes (Chen, 2007, p. 85).

The research procedures were clearly described but could have been more appropriate.

The treatment was done on only one module for one week. This seems too short for students to

get used to the use of advance organizer to benefit from it. However, the research procedures

were very clearly described so the research could be easily replicated with different subjects with

different situations. The research procedures including what, when, and how are depicted in a

clear and detailed manner in the procedure section of the study. The advance organizers and the

test items used in the study are included in the study as well.
A CRITICAL ANALYSIS OF CHENS RESEARCH 6

Research Results

It seems that appropriate statistical techniques were used in the study. This study is a

posttest only randomized experiment. Also, as explained in the study, statistic procedures,

including descriptive analysis, one-way analysis of variance (ANOVA), and repeated-measure

regression (RMR) were performed to test the research Hypotheses (p. 65). According to

Trochim (2006), typically we measure the groups on one or more measures and we compare

them by testing for the differences between the means using a t-test or one way Analysis of

Variance (ANOVA). He also points out, the posttest-only randomized experimental design is,

despite its simple structure, one of the best research designs for assessing cause-effect

relationships.

Using the aforementioned statistical techniques, each variable in the study did not emerge

in a meaningful way from the data. A quick look at the result of posttest 1 seems to show that the

research produced a meaningful effect of AO, the treatment group (group 1) with a concept map

scoring the highest followed by the second treatment group (group 2) with a text outline and the

control group (group 3) scoring the lowest. However, when the result was analyzed using the

statistical techniques, it does not demonstrate a statistically significant AO effect among the

treatment groups and the control group. The result of the second posttest even showed the

negative effect of AO; group 3 scored the highest followed by group 1 and group 2.

Discussion of Results

The results of the data analyses generally support what the researchers conclude are the

findings of the study. The researcher describes the findings of the study to not demonstrate a

statistically significant AO effect among the treatment groups and the control group. However, in

agreement with the previous research, this study shows a positive but inconclusive benefit of
A CRITICAL ANALYSIS OF CHENS RESEARCH 7

using AOs for students short-term knowledge acquisition. The students using a concept map

consistently obtained higher learning achievements than individuals using a text outline (p. iii).

As mentioned in the previous section, students in the treatment groups outperformed on the

multiple-choice quiz of posttest 1 which measures students short-term knowledge acquisition,

but the difference was not statistically significant when analyzed using the aforementioned

statistical techniques. Also, the students using a concept map did indeed score higher than those

using a text outline in all four forms of testing taken during the research: one multiple-choice

quiz and one scenario-based test on each posttest. However, it is unclear why the researcher

excluded mention of the AO effect on students short-term knowledge application even though

students in treatment groups scored higher on the scenario test of posttest 1. Both knowledge

acquisition test (multiple-choice quiz) and knowledge application test (scenario question)

produced better result among the treatment groups than the control group, but the difference is

not statistically significant.

The researcher provides reasonable explanations of the findings. As possible reasons for

the difference of no statistical significance, the researcher lists the short duration of the

treatment, small differentiation of the assessments, and loose control of the Web-based

experiment (Chen, 2007, p. 88). This attribution seems accurate with the researchers further

explanations: (1) Students did not have enough time to become familiar with the instructional

strategy of using AOs. (2) The results of scenario essay questions had little differentiation among

students. (3) It is impossible to guarantee that students follow the procedures strictly in a fully

Web-based experiment (Chen, 2007, p. 88).

Although the researcher provides a generally accurate description of the findings of the

study, the implications for practice she drew from the study do not seem accurately based on the
A CRITICAL ANALYSIS OF CHENS RESEARCH 8

findings. The researcher mentions three points as implications for practice. First, The use of

advance organizers is a good teaching and learning practice in the context of self-paced online

learning (Chen, 2007, p. 100). The study does not demonstrate a statistically significant AO

effect, and it even demonstrates a negative long-term AO effect. Recommending something that

is not based on empirical study seems unconvincing. As a second implication, the researcher

poises, The results of this study suggest that integration of advance organizers for online student

remedial programs may be beneficial (Chen, 2007, p. 101). She is drawing this conclusion

based on the study where the subject population is of high learning ability, adequate prior

knowledge of the subject matter, and sufficient experience in online learning (Chen, 2007, p.

91). Also, the scores of the quiz 2 among low-scorers are higher than those of the quiz 1

contrary to what is shown among high-scorers. Considering the time effect between the first and

the second quiz, achieving higher scores on the second quiz four weeks after the treatment seems

improbable in a reliable test setting. The conclusion drawn from this kind of seriously unreliable

test is unconvincing. The third implication the researcher drew is advance organizers may be

helpful Web-based learning devices for new online learners (Chen, 2007, p.102). As a ground

of this implication, the researcher presents nothing but a comment from one interviewee.

Summary

This study has value in that it is the first one to test the effect of AO in a fully online

learning environment. Although the study fails to demonstrate a statistically significant effect of

AO and make convincing recommendations for practice, the study provides some implications

for future research. First, the long-term effect of AO could be further studied with more reliable

instruments. The study generated a negative effect of AO on long-term. It is unclear whether

such an effect was resulted from weak test reliability or the real effect of AO. Second, future
A CRITICAL ANALYSIS OF CHENS RESEARCH 9

research could be replicated with a larger population and longer treatment duration to test an

effect of AO which is statistically significant. Third, the subject population needs to have more

diversity in learning abilities to test the effect of AO on students with different learning abilities.

Finally, a participatory organizer could be incorporated in future research to compare its effect

with that of other types of advance organizers. The use of participatory organizers is considered

effective according to the generative learning hypothesis, yet an empirical research evidence is

needed.
A CRITICAL ANALYSIS OF CHENS RESEARCH 10

References

Chen, B. (2007). Effects of advance organizers on learning and retention from a fully Web-based

class. University of Central Florida Orlando, Florida.

Trochim, W. M. (2006). The Research Methods Knowledge Base, 2nd Edition. Internet WWW

page, at URL: <http://www.socialresearchmethods.net/kb/> (version current as of October

20, 2006).

Urschel, J. D., Goldsmith, C. H., Tandan, V. R., & Miller, J. D. (2001). Users guide to

evidence-based surgery: how to use an article evaluating surgical interventions. Canadian

Journal of Surgery, 44(2), 95-100