Running head: EFFECTIVENESS OF COMPUTER BASED TESTING 1

Effectiveness of Computer Based Testing

Mia Ayres

Steven M. Bradbury

Sara K. Maglov

Matthew T. Sheehan

Loyola University Maryland

EFFECTIVENESS OF COMPUTER BASED TESTING 2

Effectiveness of Computer Based Testing

Introduction

Overview of Educational Issue

The use of computer-based assessments in place of traditional paper-pencil assessments

is on the rise in many educational settings. Over time, the number of students enrolled in schools

has increased and as a result the need to be able to assess more efficiently has risen as well

(Karay, Schauber, Stosch & Schuttpelz-Brauns, 2015). Simultaneously, technology has evolved

to provide teachers and course administrators with more options to meet the needs of a growing

population while increasing efficiency.

The use of computer based assessments has clear cut advantages for educators. The use

of computer based assessments provides teachers with a more timely way to administer, review

and grade assessments (Karay, Schauber, Stosch & Schuttpelz-Brauns, 2015). However, the

benefits for students may not be as straightforward. While today’s students have an increased

ability to utilize technology for a variety of purposes, the impact on students and their

performance, when replacing paper-pencil assessments with computer-based assessments, is up

for debate. There is also a need to explore how students feel or perceive these changes to

traditional classroom methods.

Problem Statement

There is a growing trend in the education community to use technology in the classroom

for a variety of purposes. The use of computer-based assessments is one trend that provides a

less expensive and more time efficient way to assess student understanding (Baig, et. al, 2018).
EFFECTIVENESS OF COMPUTER BASED TESTING 3

This literature review aims to consider the impact the use of these types of assessments have on

students’ performance, specifically their achievement scores. In addition, this review explores

how students perceive the use of computer based assessments in place of more traditional

achievement measures. Furthermore, relevant literature related to the benefits and the limitations

of computer-based testing will also be discussed.

Review Questions

This literature review will focus on the ways in which computer based testing impacts

student performance, defined as test scores, as well as student perceptions. This review will

address the following questions: (1) In what ways is student performance impacted by the use of

computer based assessments? (2) How do computerized tests positively affect student

performance on standardized tests? (3) In what ways can computerized assessments be limiting

for a test taker? (4) What are student perceptions regarding computer based assessments?

Purpose and Significance of Literature Review

This paper will consider the results from researching the effectiveness of computer based

tests as compared to paper-pencil tests. The effects that the different types of tests have on the

outcomes are called the “mode effects” (Oz & Ozturan, 2018). Many in the world of education

are concerned with the mode effects because the consequences impact not only the students, but

the school staff and community (Backes & Cowan, 2018). With the growing trend to increase

the use of technology in the classroom, many assessments and high stakes tests are now

administered online. Questions are surfacing about how a computer-based assessment might

impact the results of these assessments. The comparison is frequently made to paper-pencil
EFFECTIVENESS OF COMPUTER BASED TESTING 4

assessments which often preceded most of the current computer-based versions. Results are

mixed, especially when considering different content (Oz & Ozturan, 2018). The review of

literature will center on the development of three specific stances that emerge from the mixed

results: computer-based tests are more beneficial, computer-based tests are detrimental, or there

is no significant difference when comparing the results from computer-based and paper-pencil

tests.

Organization of Literature Review

This Literature Review will explore three arguments related to the use of computer-based

assessments. Some of the literature discussed will present data and information that shows that

replacing paper-pencil tasks with computer-based assessments is beneficial to student

performance and is viewed positively by student participants. Other articles reviewed in this

paper will present information that shows the use of these types of assessments are detrimental to

student performance and are perceived as negative by students. Finally, we will review articles

that show there is no significant benefit or harm to administering assessments via technology on

either student performance or student perception. The conclusion of this literature review will

synthesize the information gathered and provide answers to the review questions presented.

Literature Review

Background of Issue

Modern day education is changing drastically in an attempt to keep up with the rapid

pace of technological advances. Now, more than ever, society is pushing to integrate technology

in the classroom as much as possible. At the same time, standardized tests remain a prominent

factor in education. Many schools around the world are administering computerized versions of
EFFECTIVENESS OF COMPUTER BASED TESTING 5

these tests. Computer Based Tests (CBTs) are predicted to become the primary testing mode of

the future (Ghaderi, 2014). As of 2016, two dozen states administered tests online while most

other states are work toward transitioning to this mode (Backes & Cowan, 2018). While digital

assessments offer many advantages, such as faster score return, not much research has been

conducted to prove if computerized tests are actually beneficial to student performance in

comparison to paper and pencil based tests. The purpose of this paper is to summarize all

arguments on this topic that currently prevail in scholarly literature. Opinions on the topic vary

drastically. Professionals have argued that CBTs are immensely beneficial to students while

others seem to think CBTs can be detrimental to student success. At the same time, other studies

have shown there are no significant consequences to administering CBTs versus traditional

testing methods.

Computer Based Testing is Beneficial

One way to prove that technology based assessments have a helpful impact on

individuals is that anxiety levels found among students was found to be less than those

individuals who took a paper and pencil examination (Baig, 2018). One may think that trying

something new, such as computer based assessments, may cause stress and anxiety among those

testing. This however, does not seem to be the case. Test scores in some cases, have actually

improved when taken online. In one Florida state assessment (FCAT), students performed higher

on the Math and Reading Assessments taken online versus those who used paper and pencil

(Martindale, 2005). Students from the deaf and hard of hearing population have also strived in

being able to access online assessments along with paper and pencil (Cawthon, 2015). A final

way online assessments are beneficial is virtual contexts help students in answering multiple
EFFECTIVENESS OF COMPUTER BASED TESTING 6

choice questions correctly. It also helps them better understand their own knowledge and

learning process (Schifter, 2014). These online assessments have a way of connecting to the

students needs and wants. These are the many reasons that show online assessments are

beneficial to those students taking them.

Computer-Based Testing is Detrimental

Current literature indicates the computer-based tests can, at times, be limiting in

comparison to their counterpart, traditional paper-based tests. Many factors contribute to the

creation of a valid and reliable assessment. It is crucial that all tests, regardless of format, are

fair and equitable for all test-takers. Some studies have indicated that online assessments are not

always the best way to evaluate student performance. Factors such as test design, comfortability,

literacy, and question type can result in lower test scores. The following information outlines

some of the negative aspects of computerized tests.

Studies have shown that online assessments can result in significantly lower scores

particularly in Mathematics and English Language Arts (Backes & Cowan, 2018). Reading

comprehension tests administered online can result in lower scores than traditional assessments.

In a recent study, it was found that students who took a computer based test scored 10% of a

standard deviation lower than students who took the test on paper (Backes & Cowan, 2018).

Scores on reading assessments can be lower on computerized tests than paper and pencil tests

because lengthy text can be cumbersome to read and comprehend on a screen (Kim & Huynh,

2015).
EFFECTIVENESS OF COMPUTER BASED TESTING 7

The design of computerized tests have a large impact on its reliability. Question type on

computerized tests can be narrow. Multiple choice prompts are easier to create and grade. With

that said, computer based tests may not offer as many application problems because they require

more background knowledge from students and training from instructors (Zhao, 2012). Limited

space for scratch work on computerized tests can also be restrictive. Some students have trouble

transferring their scratch work from paper to the testing response box on computerized tests

(Backes & Cowan, 2018). Students must be skillful in editing and entering data on the testing

program in order to be successful on a CBT.

Different versions of testing questions in each mode also raise a concern. One study

compared the differences of the same question offered on a paper test and a computerized test.

The paper test offered a passage of a reading while the computer based test presented the entire

text. Students taking the CBT were expected to know that they needed to scroll through the text

in order to read the passage. If a student did not comprehend that they only needed to read a

portion of the article they likely spent a lot more time on the question than students taking the

paper-test (Backes & Cowan, 2018).

Another downside to administering computer-based tests is the costly expense. A

concern for many, is how to finance these tests. Who will cover the cost of paying someone to

grade free response questions? Who will fund the money for supplying computers to schools

that are currently without? This presents a question of ethics. Is it fair to test some children

online while others in lower socioeconomic areas are not equipped with the same resources

(Zhao, 2012)? This idea leads us to the most alarming drawback of administering tests online.
EFFECTIVENESS OF COMPUTER BASED TESTING 8

The largest flaw in computer-based tests is that a technology gap exists. Students who

are not exposed to computers regularly are presented with a gross disadvantage than students

who have access to technology all of the time. Low income and minority families are not as

likely to have computers and high speed internet readily available at home (Backes & Cowan,

2018). Urban schools are also less likely to provide digital resources. Students must be exposed

to the technology for practice in order to be successful and feel comfortable using the tools to

take an assessment. Rather than being assessed on content knowledge, students are penalized for

their lack of technology literacy (Tempel, 2012).

Even when students are adequately exposed to technology, it is possible that not all

educators have been appropriately trained to use digital tools in the classrooms. Teachers may

not have the time or expertise to determine which software or web-based application is

appropriate and compatible (Martindale, 2015). Student success can be hindered if the teacher

has not provided sufficient practice with technology in preparation for the CBT. This factor can

contribute to an unreliable assessment if an educator selects inappropriate methods for testing

prior to state mandated assessments.

Student perception regarding computerized tests often varies amongst individuals.

Negative views concerning online assessments are typically associated with test design and

comfortability levels with testing platforms. A major factor contributing to student perception is

the opportunity to review testing materials. CBTs that do not allow students to return to previous

questions are not deemed favorably by examinees (Trisnawati, 2015). Walker (2016) identifies

the rationale behind examinees who are critical of online tests. The elements of CBTs that are
EFFECTIVENESS OF COMPUTER BASED TESTING 9

considered disadvantageous to the students involved in the study include reading from a screen,

sequencing and ordering of questions, noise management of keyboards, lack of familiarity with

program, errors from typing rapidly, and not being able to see the exam as a whole (Walker,

2016).

No Significant Difference Between Testing Modes

After reviewing the literature, some research indicates that there is no significant

difference in student achievement when comparing computer-based tests (CBT) and paper-pencil

based tests (PBT). Stating that there is no significant difference does not indicate that all scores

were the exact same no matter what mode was used for testing. Instead, no significant difference

recognizes that there could have been slight differences, but those differences are slight or

insignificant. When determining that there is no significant difference between the modes,

researchers are stating that the performance from one mode could be used as a replacement for

the performance of another mode (Piaw, 2012). Other researchers summarize it by stating that

the CBT could be used as an alternative form of the traditional paper-pencil test (Oz & Ozturan,

2018).

Oz and Ozturan (2018) cite several studies from different grades and subject areas with

the same results: there is no significant difference between student scores when comparing CBT

and PBT. The subject areas that were cited are reading comprehension, English proficiency, and

math, all with the same result of no statistical difference in achievement when comparing CBT

and PBT. Within the same articles they cite other research which shows the contrary.

Ultimately their conclusion is that little research has been studied to show the reliability and
EFFECTIVENESS OF COMPUTER BASED TESTING 10

validity factors when testing students on computers and with paper-pencil (Oz & Ozturan, 2018).

The researchers sought to determine if there was reliability and validity in the scores obtained

from CBT and PBT. Their findings were that both results were reliable and had strong internal

consistencies (Oz & Ozturan, 2018).

In similar fashion, Kim and Huynh (2008) reported findings that indicate scores from

CBT and PBT are comparable. Their own study showed that the PBT scores were higher than

those from CBT, but the “magnitude of the difference was small” (Kim & Huynh, 2008, p. 567).

Hardcastle, Herrmann-Abell; & DeBoer (2017) also found some results showing differences in

early grades, but as test modes were researched in high school, no significant difference was

found between CBT and PBT. In fact, their data shows that the gap between the differences in

student performances shrinks as students move from elementary to middle school, and then

finally to high school (Hardcastle, Herrmann-Abell; & DeBoer, 2017). An earlier study by

Bennett (2003) yields similar results: the differences between modes is present, but deemed

trivial and minimal. Way, Lin and Kong (2008) cite multiple studies which also conclude that

there is “minimal mode difference.” In their own study, the researchers noted that multiple

choice tests displayed comparable results between modes, while reading and English language

arts tests where “more complicated to interpret” (Way, Lin, & Kong, 2008). Overall, their

conclusion when comparing mode effects is stated as “small or nonexistent” (Way, Lin, & Kong,

2008).

Conclusions

Summary of Educational Issue and Background

EFFECTIVENESS OF COMPUTER BASED TESTING 11

The use of computer-based assessments in place of traditional methods of assessment

such as paper-pencil tasks has proven to be a more time efficient way to determine student

learning. There is a clear benefit to the use of technology in providing quick assessment

measures and timely feedback to a large group or group of students.

When deciding if the use of computer-based assessments is also beneficial for students, it

is important to consider all sides of the issue. When considering student perceptions,

computer-based assessments have proven effective in reducing test taking anxiety thus making

students more successful (Baig, 2018). When arguing for the use of computer-based assessments

it is also important to consider accessibility. Articles reviewed suggest students with disabilities

or that require special accommodations, such as members of the Deaf or Hard of Hearing

community tend to be more successful when completing assessments in a computerized format

(Schifter, 2014).

Research that is not in support of this trend suggests inexperience with technology may

cause discrepancies in test scores (Tempel, 2012). There is also research that suggests certain

subject areas are less conducive to computer-based testing and considering this when planning

course materials can impact student success rates. (Backes and Cowan, 2018).

Furthermore, there is research that demonstrates little to no significant difference in

student performance when comparing test scores obtained through computer-based assessments

versus paper-pencil assessments. Thus arguing the more efficient computer-based assessments

are an appropriate replacement.

Discussion of Benefits to Computer-Based Testing

EFFECTIVENESS OF COMPUTER BASED TESTING 12

Student performance has increased dramatically with the use of computer based testing.

In Florida, 4th and 5th grade students results were significantly higher than those who did not

take the computer based FCAT test. Middle school students stayed the same which shows that it

does not have a negative impact (Baig, 2018). During the Pennsylvania System of School

Assessment (PSSA), all 45 students performed at a below basic level during baseline and a

proficient level on the posttest, that was taken online (Greene, 2014). Students are less stressed,

which causes low anxiety levels and students are able to perform successfully on these online

assessments. There has been increased meaningful dialogue among students who use computer

based games to prepare for online assessments. The online learning environment has a positive

effect on learning outcomes (Martindale, 2005). States and schools should take this approach of

online testing because it will ease the anxiety levels of those who experience high levels, which

would result potentially in better test scores.

Discussion of Detriments to Computer-Based Testing

Computer-based tests can, at times, be harmful to test-takers. There are two main factors

that lead to a restrictive CBT. The first is if the assessment is not properly designed. The

second, is if the assessment is administered to students who are not computer literate due to lack

of opportunities to use technology. If either of these influences are present on a CBT the

assessment could be detrimental to student performance and confidence. Student scores will be

impacted negatively, engagement will drop, and student perception will indicate a rejection for

digital assessments.
EFFECTIVENESS OF COMPUTER BASED TESTING 13

The results for this argument imply that it is crucial that the design of computer based

tests acknowledges the diverse needs of its users. During the construction of assessments,

educators must consider the background knowledge of students and more importantly their prior

experiences with technology. Students will not be successful with computer based tests if they

are not regularly exposed to programs similar to that of the testing platform. Both instructors

and students need to be trained on how to navigate the chosen testing program.

If CBTs become the only form of evaluating student progress than school systems must

make it a priority to supply all students with access to technology that connects to high-speed

internet. This is the only way to insure that online assessments will be fair and equitable for all

users, even those that come from low income families. Unless every student is equipped with the

same technological resources, computer based tests will lack validity and reliability.

Discussion of No Significant Difference Between Testing Modes

While there is evidence of the positive and negative effects of computer-based testing,

many are associated with perception and data collection methods. When comparing the actual

scores in many assessments, data appears to suggest that there is no significant difference in test

scores for students who took the same CBT as compared to the PBT. The lack of significant

difference appears to be most prevalent in the older grades, but also appears to be connected to

the subject matter and form of the test. Most studies and the literature reviews present within the

studies indicate some discrepancies with results from the research. The discrepancies

encourages the researchers to prescribe more research founded in reliable and valid data to better

understand the results of using CBT instead of PBT.

EFFECTIVENESS OF COMPUTER BASED TESTING 14

Answer to Initial Review Questions

This literature review aimed to determine what, if any, impact the use of computer-based

testing had on student performance as well as to determine how students perceive this rising

trend. The first three review questions addressed the ways in which computer-based testing

impacted student performance. We sought to discover how student performance is positively

impacted and in what ways might these assessments limit students’ performance. In this

literature review, “student performance” was defined as test scores. A review of the literature

demonstrates that test scores are on the rise in certain states and parts of the country for those

who take online assessments (Martindale, 2005). However, in instances where test scores

decline, it appears poor test creation is to blame. The literature also showed that test scores

varied depending on the different age range of the test taskers. One study suggests that younger

students may struggle to achieve on computer based assessments, but as they get older and more

familiar, scores may increase (Baig, 2018). However, some literature reviewed in this paper

indicate there is little difference in student performance when taking computer-based tests in

place of traditional paper-pencil methods. Thus suggesting computer-based assessments are a

suitable replacement for both educators and students alike.

The final review question aimed to gather information regarding student perceptions of

computer-based assessments. There is overwhelming research that demonstrates the benefits for

educators in replacing traditional paper-based assessments with technology driven options but

less on what students feel about this rising trend. Literature reviewed in this paper suggests

students are more relaxed when taking online assessments as they report lower anxiety and stress
EFFECTIVENESS OF COMPUTER BASED TESTING 15

levels (Martindale, 2005). One could argue that if students are less stressed, they will be more

engaged in the assessment and perhaps show success. Students who are showing high levels of

anxiety may score much lower. Some literature demonstrates students with little to no

technology experience or school-based technology experience feel less successful when taking

tests online (Schifter, 2014). However, students with more experience have taken the changes in

stride.

Reviewing literature related to computer-based testing and its impact on student

performance as well as exploring student perceptions on the topic has provided valuable

information which can be used by educators moving forward. Generally, by providing students

with proper exposure to technology and opportunities to practice computer-based assessments as

well as ensuring tests are well-designed, student performance can be positively impacted, or

remain relatively the same, when using these tests as a replacement for more traditional

measures. Educators should understand that comfortability with technology will play a role in

how students perceive online assessments and the more opportunities to use the computer in the

classroom in this manner the better. As we move towards developing 21st-century learners, the

use of technology in the classroom could have immediate as well as many long-term benefits for

both students and educators.

EFFECTIVENESS OF COMPUTER BASED TESTING 16

References

Backes, B., & Cowan, J. (2018). Is the Pen Mightier Than the Keyboard? The Effect of Online

Testing on Measured Student Achievement. ​Working Paper,190.​ Retrieved from

https://files.eric.ed.gov/fulltext/ED583636.pdf

Baig, W. A., Al-Zahrani, E. M., Al-Jubran, K. M., Chaudhry, T., & Qadri, A. A. (2018).

Evaluation of Test Anxiety Levels among Preparatory Year Students of PSMCHS During

Computer-Based Versus Paper-and-Pen Examination. ​International Journal of Medical

Research & Health Sciences,7(​ 1), 48-52. Retrieved from

https://www.ijmrhs.com/medical-research/evaluation-of-test-anxiety-levels-among-prepa

ratory-year-students-of-psmchs-during-computerbased-versus-paperandpen-exam.pdf​.

Bennett, R. E. (2003). ​Online assessment and the comparability of score meaning (​ ETS

RM-03-05). Princeton, NJ: ETS. Retrieved from

https://www.ets.org/Media/Research/pdf/RM-03-05-Bennett.pdf

Cawthon, S. (2015). From the Margins to the Spotlight: Diverse Deaf and Hard of Hearing

Student Populations and Standardized Assessment Accessibility. ​American Annals of the

Deaf,​ ​160(​ 4), 385-394. Retrieved from

http://www.jstor.org.proxy-ln.researchport.umd.edu/stable/26235229

Ghaderi, M., Mogholi, M., & Soori, A. (2014). Comparing Between Computer based Tests and

Paper-and-Pencil based Tests. ​International Journal of Education & Literacy

Studies,2(​ 4), 36-38. doi:10.7575/aiac.ijels.v.2n.4p.36

Greene, A. (2015). Passing Standardized Assessments With Fading Prompts. ​Journal of Special

Education and Rehabilitation,​ ​16(​ 1-2), 68-84. doi:10.1515/JSER-2015-0005

EFFECTIVENESS OF COMPUTER BASED TESTING 17

Hardcastle, J., Herrmann-Abell, C. F., & DeBoer, G. E. (2017). Comparing Student Performance

on Paper-and-Pencil and Computer-Based-Tests. 1-16. Retrieved from

https://files.eric.ed.gov/fulltext/ED574099.pdf

Karay, Y., Schauber, S. K., Stosch, C., & Schuttpelz-Brauns, K. (2015). Computer Versus

Paper—Does It Make Any Difference in Test Performance? ​Teaching and Learning in

Medicine: An International Journal,27​(1), 57-62. doi:10.1080/10401334.2014.979175

Kim, D., & Huynh, H. (2008). Computer-Based and Paper-and-Pencil Administration Mode

Effects on a Statewide End-of-Course English Test. ​Educational and Psychological

Measurement,68​(4), 554-570. doi:10.1177/0013164407310132

Martindale, T., Pearson, C., Curda, L. K., & Pilcher, J. (2005). Effects of an Online Instructional

Application on Reading and Mathematics Standardized Test Scores. ​Journal of Research

on Technology in Education,37(​ 4), 349-360. Retrieved from

https://files.eric.ed.gov/fulltext/EJ690977.pdf

Oz, H., & Ozturan, T. (2018). Computer-based and paper-based testing: Does the test

administration mode influence the reliability and validity of achievement tests? ​Journal

of Language and Linguistic Studies,14(​ 1), 67-85. Retrieved from

https://files.eric.ed.gov/fulltext/EJ1175793.pdf

Piaw, C. Y. (2012). Replacing paper-based testing with computer-based testing in assessment:

Are we doing wrong? ​Social and Behavioral Sciences,​ (64), 655-664. Retrieved from

https://www.sciencedirect.com/science/article/pii/S1877042812050549

Schifter, C. C., & Carey, M. (2014). Addressing Standardized Testing through a Novel
EFFECTIVENESS OF COMPUTER BASED TESTING 18

Assessment Model. ​International Association for Development of the Information

Society,​207-210. Retrieved from ​https://files.eric.ed.gov/fulltext/ED557369.pdf

Tempel, M. (2012). Testing Our Limits. ​ReThinking Schools,26​(3). Retrieved from

http://rethinkingschools.aidcvt.com/archive/26_03/26_03_tempel.shtml

Trisnawati, I. (2015). Validity in Computer-Based Testing: a Literature Review of Comparability

Issues and Examinee Perspectives. ​Englisia​ ​2(2). ​86-94.

http://dx.doi.org/10.22373/ej.v2i2.345

Walker, R., Handley, Z. (2016). Designing for Learner Engagement with Computer-based

Testing. ​Research in Learning Technology, 24.​ doi 10.3402/rlt.v24.30083}

Way, W. D., Lin, C. H., & Kong, J. (2008). ​Maintaining score equivalences as tests transition

online: issues, approaches and trends​. Paper presented at the National Council on

Measurement in Education, New York, NY. Retrieved from

http://images.pearsonassessments.com/images/tmrs/Maintaining_Score_Equivalence_as_

Tests_Transition_Online.pdf

Zhao, Emmeline. (2012). Online Standardized Tests are New, but are they Revolutionary?

Huffpost.​ Retrived from

https://www.huffingtonpost.com/2012/10/10/online-standardized-tests_n_1954569.html