J Sci Educ Technol
DOI 10.1007/s10956-017-9690-4
Collaborative and Competitive Video Games for Teaching
Computing in Higher Education
Spencer Smith1 · Samantha Chan1
© Springer Science+Business Media New York 2017
Abstract This study measures the success of using a col-
laborative and competitive video game, named Space Race,
to teach computing to first year engineering students. Space
Race is played by teams of four, each with their own tablet,
collaborating to compete against the other teams in the
class. The impact of the game on student learning was stud-
ied through measurements using 485 students, over one
term. Surveys were used to gauge student reception of the
game. Pre and post-tests, and in-course examinations were
used to quantify student performance. The game was well
received with at least 82% of the students that played it
recommending it to others. In some cases, game partici-
pants outperformed non-participants on course exams. On
the final course exam, all of the statistically significant
(p < 0.05) comparisons (42% of the relevant questions)
showed a performance improvement of game participants
on the questions, with a maximum grade improvement of
41%. The findings also suggest that some students retain the
knowledge obtained from Space Race for at least 7 weeks.
The results of this study provide strong evidence that a col-
laborative and competitive video game can be an effective
tool for teaching computing in post-secondary education.
Keywords Improving classroom teaching · Post-secondary
education · Programming and programming languages ·
Collaborative learning · Game design
 Spencer Smith
smiths@mcmaster.ca
1 Computing and Software Department, McMaster University,
Hamilton, Ontario, L8S 4K1, Canada
Introduction
Many studies have explored the benefits of video games
for teaching computing, but there are still aspects of this
topic that have not been investigated. For instance, many
of the previous studies have focused on teaching via tools
for game creation, as opposed to learning directly through
game play. Moreover, the previous studies are often aimed
at younger learners, not at college or university students.
Although previous studies have looked at single player
games to motivate learning through competition, fewer stud-
ies have looked at multiplayer games and the associated
opportunity for collaborative learning. Combining the above
gaps in the literature, a question emerges: can a competitive
and collaborative game be used to improve learning when
teaching computing to university students? This study aims
to address this question by measuring the use of such a game
to teach first year engineering students introductory com-
puting. Unlike many of the previous studies on using games
to teach computing, the findings will be supported by exten-
sive empirical data, based on measurements from a class of
485 students.
The class in question, ENGINEER 1D04 at McMaster
University, Canada, is an introductory programming course,
taught in the Python language. The current curriculum pre-
scribes 3 h of hands-on lab activities over the course of
12 weeks. Our experience has been that this amount of time
is insufficient for beginners. Combining this problem with a
general lack of motivation to practice programming, many
students find themselves struggling with the course content.
To motivate the students, and hopefully improve their
learning, an activity was added to ENGINEER 1D04
that combines four pedagogical ideas and technologies:

(i) educational computer games are motivational learning
tools1 (Barnes et al. 2008; Pirker et al. 2014; Prensky
2007; Barab and Dede 2007; Annetta et al. 2009; Connolly
et al. 2012); (ii) a tablet platform has the advantages of
being portable, affordable, and technologically relevant to
younger people (Baloian et al. 2013); (iii) social compe-
tition contributes to the enjoyment felt by computer game
players (Vorderer et al. 2003) and has been observed to
motivate educational computer game players (Boyce et al.
2011); and (iv) a collaborative component, through multi-
player teams, can be added to motivate those students not
influenced by competition. Empirical studies have shown
that individuals demonstrate higher levels of cooperation
within their teams when there is competition between
groups (Burton-Chellew et al. 2010). Those that are more
individualistic in nature are also more likely to contribute
under group competition, as opposed to a standard public
situation (Probst et al. 1999).
Of the ideas listed above, collaborative learning seems
particularly suited to motivating first year college students.
In an essay on approaches to improving student retention,
Tinto (2000) argues “... that colleges and universities should
make learning communities and the collaborative pedagogy
that underlies them the hallmark of the first year experience.
They should ensure that shared learning is the norm, not the
exception, of student first year experience.” With respect to
computer science education specifically, some hypothesize
that students may develop the skills necessary to succeed
in a game more readily if friends are present to motivate
and challenge them (Hicks 2010). Continuing along this
line, traditional computer science education may not be well
suited to millennial students (Nickel and Barnes 2010); col-
laborative educational games hold the promise of making
the learning experience better for both student and educators
(Nickel and Barnes 2010). As one example, collaborative
learning has been incorporating into Epik (Edutainment by
Playing and Interacting with Knowledge), an online appli-
cation for the development of quiz games (Sampaio et al.
2013). Epik is an example of one of the few serious games
that have been designed with multiplayer support. Wen-
del et al. (2013) show that collaborative multiplayer serious
games are possible and can be enjoyable, but the scope
of their study means that they stop short of incorporating
subject-specific tasks and evaluating the learning outcomes
of their game.
The current study quantifies the effectiveness of a col-
laborative and competitive video game versus a tradi-
1 The meta analysis of Wouters et al. (2013) suggests that serious
games are not actually more motivating than conventional instruction.
However, their analysis still supports the use of serious games because
their data shows an increase in student learning and retention.
J Sci Educ Technol
tional approach to teaching. The newly developed game,
Space Race, is played by teams of four, with each student
controlling their own tablet. Teams compete against one
another for the best time for completing each of the four
levels.
Using Space Race, this research provides computer sci-
ence and software engineering educators and pedagogical
researchers with insight on the feasibility and effectiveness
of a collaborative and competitive video game as a teaching
tool. As seen in the “Literature Review” section, teaching
computer programming through a game in higher education
remains a relatively new field. This study contributes the
following to this field:
– Provides evidence that a majority of students in engi-
neering would like to see video games incorporated into
education.
– Demonstrates how cooperation and group discussion
can be harnessed effectively through a game to teach
students basic computer programming concepts.
– Highlights the importance of feedback in educational
video games.
– Proves that a collaborative and competitive video game
can potentially have a positive effect on student knowl-
edge in programming immediately after gameplay.
– Shows that a collaborative and competitive educational
video game that teaches programming can positively
impact student understanding of course material.
– Reveals that programming knowledge obtained in a
video game can be used in a non-gaming context, like
on a test.
– Indicates that the understandings acquired through a
collaborative and competitive video game that teaches
programming can be retained for at least 7 weeks by
some students that played the game.
The “Literature Review” section provides a literature
review for the use of video games for teaching computing.
The design of Space Race is given in the “Game Design”
section and an overview of the experimental procedure
used to measure the impact of Space Race is provided
in the “Experimental Procedure” section. In the “Game
Reception and Feedback” section, the survey results from
the study participants are summarized and analyzed. The
“Educational Effectiveness of the Game” section mea-
sures the impact of the game on student learning, first
through a comparison of pre- and post-quiz results and
then by comparing final examination results between game
participants and non-participants. Further details on the
design and assessment of Space Race for teaching engi-
neering students introductory computing can be found in
Chan (2014), while a brief overview can be found in
Chan and Smith (2014).
J Sci Educ Technol
Literature Review
Empirical evidence is currently lacking to support game-
based learning in computing. Here, the term game-based
learning refers to the use of games as a tool to teach pro-
gramming; as mentioned previously, it does not refer to
the development of games as a method of teaching pro-
gramming. As such, many popular block-based graphical
languages such as StarLogo The Next Generation (Yoon
2005), Scratch (Maloney et al. 2004), Alice2 (Kelleher
et al. 2002), Robocode (Kumar and Khurana 2012; Long
2007), and Cleogo (Cockburn and Bryant 1998) will not be
considered here. These programming environments are not
considered games, since students have to build a game, not
play one, to learn.
Very little research has been done to explore game-based
learning as a method of instruction for programming. Fur-
thermore, most of the research that has been completed does
not definitively compare the effects of game-based learn-
ing to more traditional methods. Moreover, almost none of
the research has evaluated whether the programming knowl-
edge acquired from the game is transferable to contexts
outside of the game, and none of the games employ col-
laboration or teamwork in gameplay. Many games have
been proposed by researchers, but they have either not been
implemented or not investigated (Connolly et al. 2007).
Table 1 lists some games that have been proposed by
different researchers for implementation in software engi-
neering or computer science education. Although Chaffin
et al. (2009) use the differences between pre- and post-tests
to quantify what the game taught participants, no control
group is used to show what the students would have learned
following a traditional approach.
Table 2 lists the few conclusions that other researchers
have drawn after developing and studying the effects of a
video game that teaches programming or computing con-
cepts to students. The success of these existing projects with
educational games provides the motivation for the game
(Space Race) used in this study. To contrast with the projects
summarized in Table 2, the current project has considerably
more participants (485 students), is aimed at higher educa-
tion, incorporates cooperation and competition, includes a
statistical analysis, measures longer term impact (7 weeks
after gameplay), and evaluates the transfer of the knowledge
gained from the game to a non-gaming context (test).
Game Design
Space Race is a four-player Android tablet game that was
developed for this research project. The game has four lev-
els and is inspired by Spaceteam, a collaborative Android
and iPhone game developed by Sleeping Beast Games©
2012 (Smith 2014). When playing, each player has their
own tablet.
The game features cooperation among team members
and competition between teams. Players progress through a
level by correctly answering a series of questions served by
the game. Game performance is dependent upon the time
it takes to complete the instructions in a level; the shorter
the time, the better the performance. The best time for each
team is recorded per level on a public leaderboard that play-
ers are able to access through the main menu in the game.
The teams are listed from best to worst according to time.
Each level in Space Race introduces new programming
concepts that rely on a firm understanding of the concepts
taught in the previous levels. The programming language
being taught in all four levels is Python, version 2.7. The
emphasis is on syntax and introductory concepts. To keep
the interaction with the game simple, the decision was made
to not display tutorial style feedback for incorrect answers.
The idea being that students will give one another feed-
back or discuss the problem with the instructor. As seen in
the “Game Reception and Feedback” section, some students
would have preferred in game feedback. Each level of Space
Race was designed to be completed in an hour or less. On
average, teams completed each level within 45 min.
Level 1 Game Design
Figure 1 depicts the game screen for level 1. The player can
interact with level 1 through four touch buttons and a shake
motion. The four touch buttons are used to select answers
and the shake motion is used to indicate an Error response.
The gameplay of level 1 is very similar to that of a
multiple-choice quiz game. However, modifications were
made to ensure the design goal of cooperation between team
members is attained. Every player on the team of four will
view the same game screens. The only difference in infor-
mation being displayed is the Python data type shown in
the Panel Type display. A different data type is randomly
assigned to each player at the beginning of the game, from
the possible types of int, str, bool, or float\long.
The objective of this level is to answer all the questions in
the shortest time. All four players will see the same question
in the Question Monitor and the same four values for the Answer
Buttons. The player to select the correct answer must have
the data type that corresponds to the data type of the variable
in the question. The following is an example question:
1 X = 6
2 Y = 5
3 Z = X + Y
4 Z = ?
 J Sci Educ Technol

Table 1 Games proposed for implementation in computing education

Author(s) Game(s) Programming topics

Martin (2000) Simulation game Information systems development

Ford Jr and Minsker (2003)
Navarro and van der Hoek (2004)
Rajaravivarma (2005)
Li et al. (2008)
Muratet et al. (2009)
Chaffin et al. (2009)
“TREEZ” computer game
SimSe-Interactive Simulation Game
Number and word games
“Bomberman”
Real-time strategy game
Programming puzzles
Tree traversal techniques using iterative and recursive algorithms
Software engineering process (project management)
Basic programming skills (I/O, strings, arrays, etc.)
Introductory C programming language
Basic programming skills (I/O, strings, arrays, etc.)
Recursion

The values in the answer buttons are 10, 11, 12, and 13.
To match the correct type, the player with the Panel Type of
int must be the player to select 11. This encourages the
team to discuss the correct answer together.
In some cases, the code block shown in the Question
Monitor may raise Python exceptions, as shown below.
1 X = 6
2 Y == X
3 Y = ?
Since a Python NameError exception is raised by line
2, all four players must shake their tablets to acknowledge
the error.
The level is complete when all 45 questions have been
answered correctly. Each iteration of this level generates
the same questions in the same order. The only poten-
tial change between runs is which player should select
the correct answer, since their panel types are determined
randomly each time. Using the same questions ensures that
each team has an equal opportunity to improve their overall
time with every iteration of this level.
Levels 2, 3, and 4 Game Design
Since the underlying game mechanics for levels 2, 3, and
4 are the same, a combined description is given here. For
these levels, students are provided with a “Cheat Sheet” that
concisely summarizes each level’s concepts.
The game screen for level 2 can be seen in Fig. 2. All
of the touchscreen controls are displayed together on a sin-
gle control panel. Each touchscreen control on the panel
is paired with a touch button that is used to submit that
control’s selected value. These Submit Buttons will turn
green if the control’s submitted value is correct, red when
the control’s submitted value is incorrect, and yellow if the
incorrect control has been changed.

Table 2 Educational computing games with quantified impact on learning

Author(s) Game(s) Topics Evaluation

Kahn (1999) ToonTalk Programs, 12 pairs of students played the game and were able to complete the
algorithms puzzles. Effectiveness was not quantified.
Long (2007), Bierre et al. (2006) Robocode Java and 83 community members were surveyed. 80% of respondents per-
.NET ceived programming skills increased (Long 2007). Incorporation
in education may not be as effective because students copy open-
source code for robots online (Bierre et al. 2006).
Eagle M and Barnes (2008) Wu’s Castle Arrays For 27 students, score for the array portion of the final exam was
and loops 51% higher for game participants (p = .01).
Papastergiou (2009) LearnMem Computer LearnMem1 was compared to LearnMem2, a computer-based
memory teaching tool, with 88 students randomly split between the
concepts 2 options. No significant difference on pre-test [F (1, 86) =
2.625, p = .109]. LearnMem1 is significantly better on post-test
[F (1, 83) = 8.853, p = .004]. Survey showed students found
LearnMem1 more appealing.
Kazimoglu et al. (2012) Program Program, 25 students played the game and reported enjoying it. Effectiveness
Your Robot debug was not quantified.
Browne and Anand (2013) 6 tablet Binary search, 101 students played the games in 6 separate sessions. Quizzes
video quicksort, revealed students perform better with a combination of game and
games Dijkstra traditional instruction over either type alone. No consistent trend
algo etc. was found between game-only or traditional-only instruction.
J Sci Educ Technol
Fig. 1 Game screen for level 1
The objective for this level is for players to execute
the instructions for four different programs in the short-
est possible time. Before the game begins, each player is
randomly assigned a unique program from four available
programs. Each program is paired with a specific panel that
the player will see on their screen. This means that players
will view unique panels and program instructions on their
game screens. Table 3 lists the program and panel pairs used
for level 2. During the game, the players are served one
instruction at a time from their assigned program. A new
instruction is served when the previous instruction has been
correctly executed. Instructions indicate the value that con-
trols should be set to. Players can receive instructions for
a control on their own control panel or for a teammate’s
control panel.
In Fig. 2, player 1 has received the instruction Aileron
= 9%2. This is an example of a player receiving an instruc-
tion that affects their own panel. To correctly execute the
instruction, player 1 must set their Aileron control to the
value of 1 and press the Submit Button. Once the instruc-
tion has been cleared, player 1 will be served their next
instruction.
A player may also receive an instruction that does not
refer to a control on their control panel. For instance, player
2 may be served the instruction Message = "DAB",
when the Message control is found on player 4’s con-
trol panel, as shown in Table 3. In this instance, player
2 must verbally communicate to player 4 that Message
should be set to DAB. Once player 4 completes this task
Fig. 2 Game screen for level 2
correctly, player 2’s instruction will be cleared. Should
any player be presented with an instruction that raises a
Python SyntaxError, every team member must shake
their tablets to acknowledge the exception.
Similar to a regular program, some instructions will
require controls to be set to a new value that depends on that
control’s current value or another control’s current value. As
an example, a possible instruction a player may receive is as
follows: Feedback = Feedback + "C". The player
with the Feedback control would then refer to their Pre-
vious Values Display, as shown in Fig. 2, to determine the
current value of Feedback. This current value is the pre-
viously correct value for the control. Assuming that the
current value of Feedback is DCA, the player would have
to set Feedback to DCAC.
For research purposes, the game ensures that every player
receives the same learning experience. To achieve this, every
program has the same kind of instructions, but the instruc-
tions do not occur in the same order. For instance, Aileron
= 9%2 is instruction # 1 from Program 1 and Torque =
5%3 is instruction # 4 from Program 3. These instructions
are of the same kind; they both test the player’s understand-
ing of the modulus (%) operator. By the end of the level, all
the players will have given and received a set of instructions
that covers the same range of topics.
With the way the game is designed, it is possible for
players to finish all the instructions at different points in
time. A player will be given the instruction “Listen
to teammates!” if they have completed their programs
 J Sci Educ Technol

Table 3 Level 2 programs, panels, control names, and control types same controls, but all the controls will have unique names
except for the Terminate Loop button. Unlike level 3, all
Panel Name Type
the players will view the same file in their File Viewers.
Program 1 The game mechanics for this level are similar to levels
1 Aileron int 2 and 3, except for the addition of for-loops. The four
4 Elevator int programs a team must complete are, once again, paired
4 Rudder float/int with a panel and randomly assigned to each player at the
3 Text string beginning of the game. However, the four programs are no
2 Break bool longer separate and distinct programs. The programs follow
a pattern; instructions alternate between a level 2 or 3 style
Program 2
instruction and a group of instructions for a for-loop. The
2 Flaps int
for-loop instructions are the same across all four programs.
1 Slats int
An example is shown below.
1 Hover float/int
4 Message string 1 for Rudder in range(3):
3 Horn bool 2 Hover = Hover + 1
Program 3 3 Torque.append(Rudder)
3 Torque int 4 Roll[Rudder] = Rudder
2 Pitch int
2 Yoke float/int One control from each panel will be affected in the group
1 Speaker string of instructions. This means that all of the team members
4 Light bool will be able to participate in the execution of the for-loop.
Each line in the for-loop will be highlighted sequentially,
Program 4 starting with line 1. The purpose of the highlighter is to indi-
4 Roll int cate the line to be executed. The highlighter advances once
3 Yaw int the instruction is executed by the correct player. After the
3 Altitude float/int loop body is executed, the highlighter will loop back to the
2 Feedback string first line. Once the loop has been executed to completion, all
1 Lock bool four players must press the Terminate Loop button to indi-
cate that the for-loop has terminated. This will then prompt
the game to serve the next set of level 2 or 3 style instruc-
before everyone else. They will then have to continue to par- tions. The gameplay for the for-loop forces the players to
ticipate in the game by executing the instructions verbally traverse through the loop like a computer program.
communicated from their team members. The level ends
when all four programs have been completed.
The objective and game mechanics for level 3 are the Experimental Procedure
same as level 2. The only difference lies in the con-
trols/control types that are available and the introduction of The study was conducted with voluntary participants from
file I/O. The control types for level 3 are int, str, list of ENGINEER 1D04 in the winter term of 2013–2014. Any
int, and list of str. The layout for level 3 is shown in student from the class of 485 was free to volunteer, with no
Fig. 3. As with level 2, all the players are randomly assigned restrictions based on their characteristics (e.g., age, gender,
a program and panel pair. For this level, the program and affiliation). We did not collect data on demographic char-
panel pairs are also matched with a specific file, where the acteristics, but we did survey the students with respect to
file consists of three characters separated by newlines. The programming experience and attitudes towards video games
file I/O instructions in any given program will only refer in education (Chan 2014).
to the file that is matched with that program. This design By their self-determined decision on whether or not to
limits confusion by ensuring that a player will never have play Space Race, the students were naturally divided into two
to ask their team members for the contents of an opened groups: one control group that did not play Space Race and an
file. Furthermore, it brings the player’s focus towards under- experimental group that did. (The potential for self-selection
standing how the lines of code should be interpreted rather bias is discussed in the “Benchmark for Student Abilities”
than where the file is located. section). Since students were not required to play all four
Finally, the game screen for level 4 can be seen in Fig. 4. levels, the number of participants varied across the four
Like the previous two levels, every player will have the levels, as shown in Table 4.
J Sci Educ Technol
Fig. 3 Game screen for level 3
No limits were placed on communication between those
that played the game and those that did not. However,
diffusion (where the experimental group influences the con-
trol group) is not a significant concern since the game
was not available to students outside of the lab setting.
Moreover, if the game participants shared any of their poten-
tially increased understanding to improve the knowledge of
the non-participants, the bias would be in the conservative
direction of weakening the observed value of Space Race,
since the effect would be to reduce the performance gap
between the two groups.
The surveys on programming experience and attitude
showed that the experimental and control groups were very
similar (Chan 2014, Chapter 6). An overwhelming major-
ity of students in both groups consider themselves as novice
programmers. The students from both the experimental and
control groups had approximately the same video gaming
habits. Only 7% of the control group and 13% of the exper-
imental group had played an educational video game. Even
so, 64% of the control group and 67% of the experimen-
tal group agreed that they would enjoy playing educational
video games.
Table 5 outlines the weeks when performance data was
collected. Starting from week 3, each Space Race level
was available for students to play for 1 week. Each level
was designed to incorporate the concepts taught in the
Fig. 4 Game screen for level 4
ENGINEER 1D04 lectures, labs, and tutorials from the
previous week.
The experimental group was asked to complete a survey
(presented in the “Game Reception and Feedback” section)
after each level. Students used this survey to evaluate the
perceived usability, effectiveness, and quality of the game.
The survey allowed students to report how the game affected
their motivation to study ENGINEER 1D04. Completion of
the survey was voluntary.
Like many experimental studies, we collected perfor-
mance data by using a pre-test, followed by our intervention
(game), followed by a post-test. The difference between the
pre- and post-test measures the effectiveness of the inter-
vention. Game-based learning research, in particular, has
successfully used this technique. Some examples include an
educational game in civil engineering (Ebner and Holzinger
2007), an educational virtual reality game (Manos 2005),
and game-based learning in high school computer science
(Papastergiou 2009).
In our experiment, two identical quizzes were given to
the experimental group to complete before and after playing
a level in Space Race. The details of the quiz questions can
be found in Chan (2014). Each quiz features five multiple-
choice questions related to the material covered in the level.
Solutions to the quiz are not offered after completion of
the pre- or post-quiz. The quizzes are completed only once
 J Sci Educ Technol

Table 4 Space Race participation numbers A Kruskal-Wallis H test (Kruskal and Allen WW 1952)
was performed, using SciPy (SciPy Developers 2014), to
Level Experiment Control
compare and evaluate the results. This non-parametric test
1 236 249 was chosen because the Likert scale responses are ordinal
2 214 271 and because there are more than two categorical groups. The
3 194 291 analysis follows steps similar to the example in Opie and
4 190 295 Sikes (2004, p. 210). Kruskal-Wallis tests the null hypoth-
esis that the response distributions are equal. The p value
indicates how likely any differences observed would have
by the students. That is, the students are not asked to com- occurred if responses were drawn from the same popula-
plete the quiz again if they choose to re-play the level. All tion. The results of the Kruskal-Wallis H test, performed on
questions have to be completed before the quiz can be sub- questions 1 through 24, on the survey across the four cat-
mitted. The students were instructed to complete the quiz egorical groups are shown in Table 6. If the p value for a
individually. given response is not less than 0.05, one cannot say that
Examinations for ENGINEER 1D04 were completed by there is a significant difference in attitudes on the survey for
the entire class; therefore, the impact of Space Race can be the Likert responses between the four categorical groups.
assessed by comparing the exam performance between the However, if there is a significant result, then one can make
control and experimental groups. Since the majority of the the conclusion that at least one of the groups is different
exam questions were designed by the 1D04 instructor, not from the others. The test does not identify where the differ-
by us, the questions should have minimal bias. 1D04 used ences occur or how many differences actually exist (Kruskal
two midterm exams and one final exam to evaluate students’ and Allen WW 1952).
proficiency. All exams were multiple choice. The results of Table 6 indicate that there are significant
differences for at least one group for questions 1, 3, 6, 12,
13, and 14. Since the game mechanics of levels 2, 3, and 4
Game Reception and Feedback are very similar, while the game mechanics for level 1 are
different, the Kruskal-Wallis H test was performed again,
This section provides the results of the student survey, which but this time only with the responses for levels 2, 3, and
included Likert scale questions and written answers. An 4. The results in Table 6 indicate that for levels 2, 3, and
excerpt from the survey is shown in Fig. 5; the full ques- 4, only questions 1 and 3 have responses that are signifi-
tionnaire is available in Chan (2014). The survey results are cantly different. Going forward, all of the questions that had
used throughout this section to support the qualitative obser- insignificant differences between the four levels are ana-
vations made while watching students play Space Race. lyzed using the pooled data. In those rare cases where it is
The number of respondents for levels 1 to 4 was 180, 180, necessary, the analysis of the responses will consider the
92, and 98, respectively. The Likert response results of the differences between levels. Further details can be found in
survey (questions 1 through 24) from level 1 are shown Chan (2014).
in Fig. 6. The Likert responses for the other levels look
qualitatively similar (Chan 2014). Playability
To determine if the survey responses on the different
levels are actually similar, each question was evaluated to deter- Questions 1–5 and 24 on the Likert response portion of
mine if the results across the four levels can be combined. the survey relate to playability. A majority of the written
feedback mentioned the need for a more interactive tuto-
Table 5 Experimental timeline and course schedule rial on gameplay. One student said that they “like the game
however the instructions were not very intuitive and it was
Week no. Item
hard to initially learn how to play. After learning how to
Week 1 Classes begin play it was more fun.” An example of the current tutorial
Week 3 Space Race level 1, survey page, which is shown before each level, is shown in Fig. 7.
Week 4 Space Race level 2, survey The page features the game screen, along with explanations
Week 5 Space Race level 3, survey for each control, which students access by clicking on the
Week 6 Space Race level 4, survey, and midterm 1 appropriate button. We observed that the students rarely, if
Week 10 Midterm 2
ever, read the instructions before playing the game. This
Week 12 Classes ends
behavior may have contributed to the perceived difficulty in
Week 13 Final exam
learning how to play the game. It also suggests that the tra-
ditional method of defining game rules through text is no
J Sci Educ Technol
Fig. 5 Sample survey questions
longer effective. Perhaps, as a student suggested, “...a video
or example would be much more effective.” While a video
tutorial could be more interesting for students to review, it is
not clear whether a tutorial presented in a different manner
would have any effect. Andersen et al. (2012) show that the
value of tutorials is highly dependent upon the complexity
Fig. 6 Level 1 survey Likert response results
of the game; they have a surprisingly negligible effect on
player engagement in games that are less complex or similar
to existing games in the same genre. Furthermore, providing
help on-demand, during gameplay, can either have positive,
negative, or negligible effects on player engagement. Given
the relatively simplistic game mechanics of Space Race, a
 J Sci Educ Technol

Table 6 Survey levels 1, 2, 3,

and 4 Kruskal-Wallis H test. Levels 1, 2, 3, and 4 Levels 2, 3, and 4
Significant at p < 0.05
Quest H-stat p-val Signif? H-stat p-val Signif?

Q1 32.861 0 Yes 27.41 0 Yes

Q2 7.752 0.051 No 0.549 0.76 No
Q3 22.865 0 Yes 10.798 0.005 Yes
Q4 4.083 0.253 No 4.165 0.125 No
Q5 6.419 0.093 No 0.458 0.795 No
Q6 7.894 0.048 Yes 4.001 0.135 No
Q7 7.139 0.068 No 4.314 0.116 No
Q8 1.389 0.708 No 1.311 0.519 No
Q9 2.171 0.538 No 0.088 0.957 No
Q10 1.117 0.773 No 0.929 0.628 No
Q11 5.624 0.131 No 0.887 0.642 No
Q12 19.038 0 Yes 0.152 0.927 No
Q13 11.028 0.012 Yes 4.012 0.135 No
Q14 11.175 0.011 Yes 4.67 0.097 No
Q15 4.62 0.202 No 2.696 0.26 No
Q16 4.062 0.255 No 2.119 0.347 No
Q17 4.788 0.188 No 4.586 0.101 No
Q18 0.717 0.869 No 0.625 0.732 No
Q19 1.616 0.656 No 1.487 0.475 No
Q20 4.214 0.239 No 3.863 0.145 No
Q21 4.224 0.238 No 3.081 0.214 No
Q22 0.264 0.967 No 0.073 0.964 No
Q23 2.848 0.416 No 0.935 0.626 No
Q24 1.802 0.614 No 0.459 0.795 No

complete tutorial seems unnecessary. However, the impact
of on-demand help during gameplay should be investigated
further.
Overall, a large majority of the students enjoyed the
game, with at least 82% agreeing with the statement “I
enjoyed playing the video game” (survey question 4, Fig. 5).
Level 2 had the highest proportion of students in agreement
with the statement, at 88%. A student commented that they
“enjoyed the whole game, [because] it helped [them] learn
some basic programming techniques and cleared up some
uncertainty.” Further feedback included the suggestion that
a “storyline element [be added to the game] as it makes it
much more immersive and easier to remember when told as
a narrative.”
Teachability
Questions 6–8, 11, 12, 15–17, 22, and 23 on the survey
are the Likert responses that relate to the teachability of
the game. These questions explore students’ perceptions
of the effectiveness of Space Race for teaching basic pro-
gramming concepts. The teachability questions also explore
whether Space Race was able to motivate students to study
the course material and whether students prefer the game
over traditional teaching methods.
When students were asked if “the video game helped
[them] to learn the basic programming concepts being pre-
sented” (question 6), at least 85% students answered with
agreement in all four levels. Level 4 had both the most
students that agreed with the statement (88%) and dis-
agreed with the statement (6.5%). The Kruskal-Wallis H
test results shown in Table 6 reveal that there was a sig-
nificant difference in the results between the four groups.
However, this can most likely be attributed to the difference
in the proportion of students that chose “Strongly Agree” as
opposed to “Agree” since all four levels had approximately
the same proportion of students that chose “Neither Agree
nor Disagree” and “Disagree”/“Strongly Disagree”; all were
comparatively low. Notably, level 3 and level 4 had almost
double the proportion of students that selected “Strongly
Agree” versus level 1.
For question 8, at least 82% of the students felt that they
“would recommend that others try to learn the basic pro-
gramming concepts with [Space Race]” for all four levels.
J Sci Educ Technol
However, from the written feedback, a student noted that
“the game confirms knowledge [; it] doesn’t really teach.”
Some other students expressed the same feelings and felt
that Space Race needed “a better response system to incor-
rect answers.” For example, a student suggested that “hints
[appear] if a team continuously got a certain question
wrong.” To summarize, students felt that the game needed
more feedback to teach them concepts, instead of simply
confirming correct knowledge. Although the game itself
does not directly teach, it certainly does create multiple
opportunities for team members to teach each other. This
will be discussed further in the “Cooperation” section.
Furthermore, the game pushes students to reference their
“Cheat Sheets” in levels 2, 3, and 4 for assistance and feed-
back. Students felt that “the cheat sheet was very helpful in
summarizing the game.”
From our observations, although the game does not
directly teach new concepts, it does create many “teachable
moments.” An instructor was always present during game
play to answer student questions. This allowed the instruc-
tor to understand where the students needed to improve.
Moreover, the game facilitated the students identifying the
concepts they were not familiar with. The game encour-
ages students to ask questions. A student is perhaps more
likely to receive help from an instructor when they are work-
ing with team members, because a less assertive student’s
more assertive teammates will make sure that the ques-
tion is asked. Also, there is less shame associated with not
understanding a concept if a student’s peers also do not
understand. This, in turn, could lower a student’s anxiety
towards approaching their instructor, an authority figure, for
Fig. 7 Space Race level 3
tutorial paage
help. Finally, asking an instructor for help in the context
of a game could perhaps be perceived as lower risk for the
student than asking a question in another context, such as
during lectures, tutorials, or labs.
Question 15 was used to assess students’ preference for
an educational video game over traditional teaching meth-
ods. At least 65% of the students agreed with the following
statement: “I would prefer to learn basic programming con-
cepts through a video game as opposed to other traditional
teaching methods such as lecture tutorials, or textbooks.” In
the written response section, a student also said that “when
[they] played this game [they] felt as if [they] were studying.
[They] would enjoy using this game as a break from the reg-
ular studying for tests and exams.” Another student stated
that they “learned a lot from this game as [they] were able to
apply what [they learned] in lectures and tutorials.” Finally,
a student said that “the interactions between team members
bolstered the understanding of concepts and created a better
learning experience than just reading a textbook.”
The results of questions 16 and 17 can be used to gauge
how effective Space Race was in motivating students to
be more interested in the subject of programming. When
students were asked if “the video game stimulated [their]
curiosity on the basic programming concept being pre-
sented” (question 16), at least 74% agreed and at most 7.4%
disagreed with the statement for all the levels. Furthermore,
at least 67% agreed that “the video game has motivated
[them] to review the course material” across all four levels.
Therefore, for some students, the game has the side effect of
motivating them to seek an understanding of programming
outside the context of the game.

Cooperation
Perhaps the most important aspect of Space Race is the
incorporation of cooperation into the game. Questions 9,
10, 13, 14, 18, and 21 provide insight on how coopera-
tion affected student learning. From question 9, across all
levels, as little as 2.3% and at most 7.4% disagreed with
the statement that “[their] team members helped [them] to
better understand the basic programming concepts being
presented.” Furthermore, only 0 to 2.3% disagreed with the
statement that “cooperation between team members makes
this video game an effective teaching tool for learning basic
programming concepts” (question 18) for all the levels.
From this, one can conclude that cooperation played a large
part in the effectiveness of Space Race as a teaching tool.
Students have also provided the following feedback with
regard to cooperation, which shows that they used their
teammates to facilitate learning:
– “I really enjoyed the cooperative aspects of the game
because it does a tremendous job of encouraging team-
work and sharing individual knowledge for the benefit
of a common goal....”
– “When we were stuck on a question, the team would
offer suggestions and we would get through the ques-
tion together. This was helpful when questions were
hard and especially when the player did not know the
correct answer on their own.”
– “[My team members] helped me a lot, as sometimes my
initial guess was incorrect, but they [would] correct me
and explain the concept.”
– “Communicating with team members encourages dis-
cussion of concepts which I like.”
– “I really liked the cooperative nature of this video
game. Individually, we may have struggle [sic] with the
concepts; but as a team, we conquered them.”
The individuals in each group had different levels of
understanding computing. From the feedback listed above,
it is clear that the less informed students felt they were able
to learn from their more capable peers. For their part, the
more capable peers also seem to have generally benefit-
ted from the experience. From the comments received on
the survey, most of the students that felt confident with the
course material still enjoyed the game; they felt a sense of
pride in being able to instruct others. For example, a student
stated that they “[were] the lead of the team so [they] did not
learn anything from [their team members] however, [they]
taught them a lesson or 2 [sic] about Python.” It is important
to note that there were very few comments on the survey
where students felt their team members did not teach them
anything new. Some of the “negative” comments could even
have a positive interpretation. For example, a student said
that “[their teammates] slowed [them] down but [it] was fun
J Sci Educ Technol
to play with them.” Another said “[their teammates were]
great but they need to be more efficient.” It was much more
common for students to comment that their team members
were able to teach them a concept they did not understand.
To quantify, across all levels, only 9 out of 231 (0.039%)
responses for question 26, “How did your team members
affect your ability to learn the basic programming concepts
being presented,” could be interpreted as being “negative.”
The positive attitude towards cooperation in Space Race
may be partly because the interaction between teammates is
face-to-face, as opposed to being virtual, as is the case for
many other video games. The benefits of face-to-face inter-
action are supported by the success of Augmented Reality
games for teaching science (Dunleavy et al. 2009; Cheng
and Tsai 2013). As Dunleavy et al. (2009) observe, in-
person interaction has a much greater bandwidth on multiple
dimensions when compared to Multi-User Virtual Environ-
ments (MUVEs). Moreover, instructional conversation, like
that between teammates explaining concepts to one another,
is generally believed to provide learning benefits, especially
when the talk is interpretive; that is, the greatest learning
gains come when the conversation is generated in the service
of analysis or explanation (Palincsar 1998). Players work-
ing in groups was also shown to be beneficial for learning in
the meta-analysis of the cognitive and motivational effects
of serious games (Wouters et al. 2013).
Educational Effectiveness of the Game
This section will quantify the educational effectiveness of
Space Race. Specifically, the pre- and post-quizzes will
be analyzed, followed by the ENGINEER 1D04 exam
results. The quiz results demonstrate the immediate effects
of Space Race, while the exam results demonstrate whether
the knowledge gained can be retained over time. The exam
results also allow consideration of whether the understand-
ing gained through Space Race can be transferred outside of
the game.
We use the conventional value of p < 0.05 as our test
for statistical significance, even though we have multiple
tests. In cases with n tests, the Bonferroni adjustment sug-
gests lowering the p value to 0.05/n to account for the
increased chance of getting an incorrect “significant” result
(Type I error) (Bland and Altman 1995). However, we have
not used the Bonferroni correction because it is too conser-
vative, being concerned with the universal null hypothesis,
which states that all null hypotheses are true simultaneously
(Perneger 1998). The Bonferroni correction has the logical
weakness that the results will change depending on the num-
ber of tests. Moreover, the Bonferroni correction increases
the likelihood of type II errors (Perneger 1998). Rather than
use statistics to make sense of the combination of the test
J Sci Educ Technol
results, we present the statistical results separately for each
test and then use common sense to interpret them. Rather
than think of our approach as multiple tests of the same
hypothesis, we think of it as a series of individual tests,
each with their own hypothesis. Our null hypothesis is not
that “the proportion of students that are correct do not differ
between the experimental and control groups for the exam-
ination”, but rather that “the proportion of students that are
correct do not differ between the experimental and control
groups for Question X.”
Benchmark for Student Abilities
Since members of the experimental group were volunteers,
we need to first assess that self-selection bias is minimal
before causation can be attributed to Space Race. A method
of benchmarking must be employed to determine the distri-
bution of students in the experimental and control groups.
To do this, a combination of each student’s math and physics
exam marks were used. Both of these exams are manda-
tory for all first year engineering students at McMaster.
Almost all of the study participants will have taken these
exams in the term preceding our experimental study. Math
and physics were chosen because research has shown that
some misunderstandings arise from parallel or identical
causes between the domains of mathematics, physics, and
computer programming (Perkins and Simmons 1988).
Each student was assigned a score S, which is calculated
using the formula
S = Average(M, Average(P1 , P2 )), (1)
where M, P1 , and P2 are the grades on the mathemat-
ics final exam, physics midterm 1, and physics midterm 2.
When using this equation, the following rules were applied
to special cases:
1. If only one of P1 or P2 was available, the available mark
was used.
2. If only one of Average(P1 , P2 ) or M was available, the
available mark was used.
3. If no marks were available, the student was omitted
from the evaluations.
Following these rules, five students could not be assigned
a score. Table 7 shows the number of students omitted for each
level. The number of participants eliminated is much smaller
than the size of the original sample they were removed from.
Therefore, the elimination of participants is considered to
have a negligible effect on the observed results.
Visual comparison of the score distributions between the
experimental and control groups suggests that these groups
can be considered as drawn from the same population. To
confirm this, a two-sided t test was performed for each
level to determine whether the differences between the two
groups are insignificant. Since the t test assumes the sam-
ples are normally distributed, the normality test from SciPy
was used to first verify that both the experimental and con-
trol groups are normally distributed. The results are shown
in columns 4 and 5 of Table 7. Given that all the p val-
ues are larger than the level of significance of 0.05, the null
hypothesis that the samples come from a normal distribution
cannot be rejected. As such, the distribution of scores for
both the experimental and control groups are assumed to be
normal.
Having established normality for the distribution of
scores for both the control and experimental groups, the t
test was performed with the null hypothesis that the two
independent samples, the experimental and control groups,
have the same distribution. The results of the t test are shown
in the last two columns of Table 7. It can be seen that the
p value for all four levels is well above 0.05. Consequently,
the null hypothesis cannot be rejected. This suggests that
there is very weak evidence that the distribution of scores
between the experimental and control groups are not the
same, which implies that self-selection bias is not a con-
cern. Therefore, differences in performance between the two
groups can reasonably be attributed, at least in part, to the
influence of Space Race.
Pre- and Post-quiz Results
As mentioned previously, in the “Experimental Procedure”
section, the pre- and post-quizzes for each level were iden-
tical. An example quiz question, for level 1, question 5
(L1Q5), is shown in Fig. 8. The full set of quiz questions
are available in Chan (2014). Students were not told which
questions they answered correctly.
Table 8 shows the percentage of students that answered
correctly on the pre- and post-quiz. The table also includes
the number of students that successfully completed the pre-
and post-quiz questions for each level (indicated by N). This
number may be less than the number of students that actu-
ally played and completed that level of Space Race, since
some students were unable to submit the post-quiz due to
technical issues. The quiz data, both pre and post, for these
students has been eliminated from the results.
McNemar’s test was chosen to verify the significance of
the differences between the pre- and post-quiz results for
each question. McNemar’s test can be used when the data
being investigated is paired and nominal and the outcome of
interest is a proportion (McNemar 1947). The null hypoth-
esis in this instance is that the proportion of subjects that
answer a question correctly is the same before and after play-
ing Space Race. The results of McNemar’s test can be seen in
Table 8. The table shows that 65% (13/20) of all the
results are significant. Additionally, all of the results that are

Table 7 Experimental and
control group statistics Eliminated
Lvl Experiment
1 2 of 236
2 2 of 214
3 2 of 194
4 1 of 190
significant are cases where the proportion of students that
were correct in the post-quiz exceeded that of the pre-quiz
(positive delta correct). On the other hand, all of the negative
delta correct results were relatively small and statistically
insignificant.
Level 1 had two questions where the difference was not
significant: questions 2 and 3 (abbreviated to L1Q2 and
L1Q3). The insignificance of the results suggests that there
was no change in the proportion of students that were cor-
rect before and after game play. Both questions required
students to generalize what they observed in Space Race.
The game mechanics of level 1 make it easy for students to
simply guess until they got the right answer. Moreover, the
game does not provide feedback for incorrect answers. This
combination could have resulted in the poor results on L1Q2
and L1Q3.
Out of the four levels, students improved the least on the
level 2 quiz, with only one of the positive delta correct cases
being significant. L2Q1 saw negative improvement with a
delta of −1.1%. However, the percentage of students that
were correct before and after playing Space Race is over
95%. This suggests that the question was perhaps too easy;
the concept it tested was already understood by the large
majority of students prior to game play.
L2Q2 did not show a significant improvement in perfor-
mance after the students completed the game. L2Q2, shown
in Fig. 9, was used to assess whether students understood
that the value of a variable (of an immutable type) cannot
change when the value is used, only when a new value is
assigned to the variable.
The topic of variable assignment was handled in Space
Race with instructions like “Yoke = float(7/2).” The
student must change their Yoke control to 3.0 because a
value has been assigned to it. Without assignment, none of
Fig. 8 Sample quiz question, level 1, question 5 (L1Q5)
J Sci Educ Technol
Normality Tst p-val t test
Control Experiment Control t-stat p-val
3 of 249 0.421 0.595 0.5214 0.6024
3 of 271 0.364 0.699 1.2440 0.2142
3 of 291 0.081 0.669 1.0046 0.3157
4 of 295 0.355 0.668 0.8312 0.4064
the values for their controls would change. Unfortunately,
students appear to have been unable to abstract their under-
standing from Space Race and apply it to L2Q2. Perhaps,
L2Q2 would have met with more success if there was an
explicit example in the game where an instruction does not
cause any controls to change because assignment has not
taken place. Providing in-game feedback may also have
been helpful. The game could have explicitly revealed to the
player that changing control values because of an instruc-
tion is the same as changing a variable’s value through an
assignment statement.
Table 8 shows insignificant improvement for L2Q3,
which is surprising because of its close relationship with
L2Q5. L2Q5 asks students to identify, from a list of types,
which type is not mutable. Eighty-five percent of the stu-
dents were able to correctly identify strings as the
correct answer after playing Space Race, a 16% improve-
ment from the pre-quiz. However, only 31% of the students
were able to select the correct answer for the seemingly
similar question L2Q3, shown in Fig. 10.
L2Q3 (Fig. 10) requires students to identify the option
that produces an error; the correct answer (A) is the one
where the line of code incorrectly attempts to change the
string via: “X[0] = "E".” The results from L2Q5 and
L2Q3 suggest that students know strings are not mutable,
but they do not understand what this implies. All of the
options from L2Q3 appear in the questions for level 2. That
is, all of the incorrect answers for L2Q3 are instructions that
students have to execute in the game and the correct answer
is an “Error” result that they would have faced. The prob-
lem might be traced to the use of Python’s slice notation
in L2Q3, since the students were observed to struggle with
this notation. The inability to recognize the “Error” option
in L2Q3 as the correct answer may perhaps highlight Space
Race’s failure to teach students to identify exceptions. Feed-
back from the game to explain why their error was cleared
might be beneficial.
Table 8 shows that levels 3 and 4 had the greatest num-
ber of significant results. The only insignificant result is for
L3Q4 which, like L2Q1, had a high proportion of students
correct for both the pre- and post-quiz. L3Q5 (Fig. 11) had
the highest improvement out of all the quiz questions, with
a delta correct of 65%. This question is almost identical
J Sci Educ Technol

Table 8 Pre- and post-quiz

McNemar’s test results. Quest Pre (%) Post (%) Delta (%) Chi-Sqr p-val Signif?
Significant p < 0.05
Level 1, N = 230
1 17.83 40.87 23.04 44.59 0 Yes
2 26.09 30.43 4.35 1.79 0.181 No
3 71.3 69.13 −2.17 0.36 0.547 No
4 39.57 46.96 7.39 6.42 0.011 Yes
5 19.13 60 40.87 83.36 0.000 Yes
Level 2, N = 188
1 96.28 95.21 −1.06 0.33 0.564 No
2 23.94 28.72 4.79 3.24 0.072 No
3 26.6 30.85 4.26 2 0.157 No
4 53.19 54.79 1.6 0.36 0.549 No
5 69.15 85.11 15.96 22.5 0.000 Yes
Level 3, N = 171
1 66.67 73.68 7.02 5.14 0.023 Yes
2 30.41 46.2 15.79 13.75 0.000 Yes
3 66.08 81.29 15.2 16.1 0.000 Yes
4 90.06 90.64 0.58 0.04 0.835 No
5 17.54 82.46 64.91 98.57 0.000 Yes
Level 4, N = 167
1 79.04 92.81 13.77 17.06 0.000 Yes
2 61.08 69.46 8.38 6.53 0.011 Yes
3 49.1 59.88 10.78 9 0.003 Yes
4 72.46 85.03 12.57 13.36 0.000 Yes
5 31.14 61.68 30.54 40.02 0.000 Yes

to one of the instructions that appears in the programs of
level 3.
Even though this concept was explained on the “Cheat
Sheet,” many students struggled with this instruction. This
is not surprising, since the percentage of students that got
L3Q5 correct (answer C) in the pre-quiz (18%) was the low-
est of all the questions. While playing the game, students
would often require the instructor’s help on this point. This
question is different from the other instructions because it
is less intuitive, since the empty string ("") is part of the
answer. This provided the instructor with an opportunity
to explain the concept in more depth. This suggests that
the game may be most effective when it motivates inter-
action with an instructor on a topic that requires deeper
explanation.
Fig. 9 Question L2Q2
Comparing the Effects of Space Race on Different
Students
The next question we asked was whether student ability had
an influence on the degree of improvement in the post-test
results (Chan 2014). To compare the effects of Space Race
on “stronger” and “weaker” students, all the students within
the experimental group were sorted into three separate bins
based on their S value (Eq. 1):
– Bin 0: S < 50%
– Bin 1: 50% ≤ S ≤ 75%
– Bin 2: S > 75%
The students in Bin 0, Bin 1, and Bin 2 represent the
weakest, average, and strongest students, respectively. The
analysis from the previous section was run on each bin
to determine the delta in correct answer on the post-quiz,
using the p value to judge significance (Chan 2014). Results
show that 6/20 (30%), 11/20 (55%), and 10/20 (50%) of the
improvements are significant for Bins 0, 1, and 2, respec-
tively. This indicates that Space Race had a larger positive
impact on the average and stronger students than on the
weaker students. Also, it appears that the impact of Space
Race on stronger and average students are approximately
the same.

Fig. 10 Question L2Q3
Midterm Exam Results
This section compares the performance of the experimental
and control groups on the multiple-choice midterm exams.
We selected the ten questions from midterm 1 and the two
questions from midterm 2 that covered the same concepts as
Space Race. Each selected question was matched with one
or more game levels. This was necessary to determine which
students belong to the experimental or control groups, since
not everyone that participated completed all of the levels.
Table 9 shows the proportion of students that were cor-
rect in both the experimental and control groups for each
question of the midterm exams. The assigned levels for each
question are listed, along with the delta correct value, which
shows the difference in percentage of students that were
correct between the experimental and control groups.
The chi-square statistic was chosen to test the signifi-
cance of each result. This statistic can be used to investigate
whether the distribution of the categorical variables differ
from one another (Moore 1976). It compares the counts
of categorical responses between two or more independent
groups. The null hypothesis posits that the proportion of
students that are correct do not differ between the experi-
mental and control groups. Table 9 shows the results of the
chi-test.
The results show that 4/12 (33%) of the questions had
a significant difference in performance between the exper-
imental and control groups. All of these significant results
are for a positive delta correct value, with the largest
improvement being 15% for question 27 on midterm 2.
That is, all of the significant results are represented by
cases where the experimental group outperforms the control
group. The one negative delta correct result was not statisti-
cally significant. From this, one can conclude that students
from the experimental group either perform equally well or
Fig. 11 Question L3Q5
J Sci Educ Technol
better, than the students from the control group. One pat-
tern that can be drawn from the results in Table 9 is that
level 1 did not result in a significant positive delta correct
value.
Final Exam Results
The final exam results are presented in the same manner as
the midterm 1 and 2 results in the “Midterm Exam Results”
section. The chi-square statistic was again used to test the
significance of the results, as shown in Table 10. The inter-
pretation of the final exam results differs from the midterms,
because the final occurred at a later time, as shown in
Table 5. At least 7 weeks passed between each student’s last
exposure to Space Race and their completion of the final
exam. Therefore, the performance of the experimental group
on the final exam can indicate whether the students were
able to retain the concepts they learned in Space Race over
a longer time period.
Questions 5, 6, and 7 on the final exam were set by us,
not by the instructor. These questions are identical to the
pre- and post-quiz questions given in Space Race: question
5 from L3Q5 (Fig. 11), question 6 from L2Q3 (Fig. 10), and
question 7 from L1Q5 (Fig. 8).
Table 11 shows a comparison of how students that com-
pleted the post-quiz for each relevant question performed on
the exam. For example, for question 5 (L3Q5), 141 students
had the correct answer on their post-quiz; of those 141 stu-
dents, 114 students (about 80%) had the correct answer on
their final exam. Alternatively, 30 students got question 5
(L3Q5) incorrect on their post-quiz, but 15 of them (about
50%) got the correct answer on the final. The results show
that only 55% of the students that had the correct answer
on their post-quiz for question 6 selected the correct answer
again on the final exam. However, questions 5 and 7 showed
significantly better retention rates, with 80% of the students
maintaining their correct answer. Not all of the students
retained their knowledge after playing Space Race, but the
majority did. In particular, questions that saw a large delta
correct from pre- to post-quiz results had a higher retention
rate. In all cases, some students that originally selected the
incorrect answer on the post-quiz chose the correct answer
on the final. Space Race may have been a contributing factor
to motivating them to study the topic being tested.
J Sci Educ Technol

Table 9 Midterm chi-square

test results. Significant if Quest Level Expt correct (%) Control correct (%) Delta correct (%) Chi-square p-val Signif?
p < 0.05
Midterm 1
1 1 54.69 50.00 4.69 0.64 0.423 No
5 1, 2 89.58 93.33 −3.75 1.22 0.270 No
9 1 96.35 94.44 1.91 0.40 0.527 No
10 4 46.75 36.24 10.51 3.71 0.054 No
13 3 51.92 46.76 5.16 0.77 0.380 No
17 4 74.03 63.30 10.72 4.27 0.039 Yes
20 3, 4 35.26 29.63 5.63 1.07 0.300 No
23 2, 3, 4 59.30 47.50 11.80 4.71 0.030 Yes
27 3, 4 70.51 63.43 7.09 1.73 0.188 No
28 3 70.51 57.87 12.64 5.69 0.017 Yes
Midterm 2
5 1 78.80 75.14 3.67 0.50 0.479 No
27 3, 4 45.70 30.37 15.32 8.30 0.004 Yes

Table 10 Final exam

chi-square test results. Quest Level Expt correct (%) Control correct (%) Delta correct (%) Chi-square p-val Signif?
Significant if p < 0.05
4 1 51.83 51.08 0.76 0.00 0.965 No
5 (L3Q5) 3, 4 77.56 36.20 41.37 61.22 0.000 Yes
6 (L2Q3) 2, 3 48.26 32.68 15.57 8.83 0.003 Yes
7 (L1Q5) 1 80.63 63.98 16.65 12.26 0.000 Yes
11 4 88.16 82.67 5.49 1.73 0.189 No
16 1, 2 84.82 84.95 −0.13 0.01 0.913 No
17 1 37.70 43.55 −5.85 1.11 0.293 No
21 4 84.87 87.11 −2.24 0.22 0.640 No
23 3, 4 82.69 79.19 3.51 0.51 0.474 No
29 3, 4 76.28 61.54 14.74 8.42 0.004 Yes
44 3, 4 33.33 21.27 12.07 6.27 0.012 Yes
46 1 64.92 67.74 −2.82 0.22 0.638 No

Table 11 Final exam results

compared to post-quiz results Post-quiz correct sample Post-quiz correct Exam correct Exam/post-quiz
Quest 5 N = 171 141 114 80.85%
Quest 6 N = 188 58 32 55.17%
Quest 7 N = 230 138 111 80.43%
Post-quiz incorrect sample Post-quiz incorrect Exam correct Exam/post-quiz
Quest 5 N = 171 30 15 50%
Quest 6 N = 188 130 51 39.23%
Quest 7 N = 230 92 70 76.09%

Questions 5, 6, and 7 had the largest difference in per-
formance between the experimental group and the control
group as seen in Table 10. Out of these questions, question
5 had the biggest delta correct value (41%). Interestingly,
question 5 (L3Q5) also had the greatest delta correct value
for the pre- and post-quiz as seen in Table 8. This could sug-
gest that most of the students that did not play Space Race
failed to learn the concept tested in question 5.
Overall, 5/12 (42%) of the results from Table 10 were
significant. None of the negative delta correct values were
significant results. This, again, suggests that students that
played Space Race either performed equally well or better
than the students that did not play Space Race. For questions
5, 6, and 7, it is unclear whether the experimental group per-
formed better than the control group simply because they
had seen the questions before, although it is important to
note that the post-quiz did not provide any feedback to
students, i.e., students were not told whether their answer
was correct or incorrect, unless they sought the answer out
themselves.
Concluding Remarks
This paper has explored the effectiveness of a collaborative
and competitive video game in improving student under-
standing of basic programming. Based on the results of
this study, game-based learning can effectively be used to
teach basic computer programming concepts to students in
higher education. In particular, cooperation within a game
world can be used effectively to encourage students to teach
one another. Also, in some cases, the knowledge learned in
the game can be transferred to a non-gaming context and
retained by the student.
As proposed in the “Introduction” section, the combi-
nation of pedagogical idea and technology used in this
study were successful in motivating students and improv-
ing their performance. The specific ideas that are supported
by the findings include the following: (i) educational com-
puter games can be used as motivational learning tools; (ii)
a tablet platform can be used effectively for educational
games in higher education; (iii) social competition can con-
tribute to the enjoyment and motivation felt by computer
game players; and (iv) a collaborative component, through
multiplayer teams, can be added to motivate those students
not influenced by competition.
This study demonstrated that students that played Space
Race were able to outperform students that did not play
Space Race on course exams in some instances. However,
it does not directly compare the educational effectiveness
of the video game to more traditional methods of teach-
ing. One could argue that gaming participants did better in
J Sci Educ Technol
some cases simply because they spent more time on com-
puting, since playing all four levels of Space Race required
about an hour per week for 4 weeks. A future study should
look at whether non-gaming participants that spent the same
amount of time with traditional methods of learning, such
as reading a textbook or attending a tutorial, would per-
form equally well or surpass the performance of the gaming
participants.
Work still remains to gain a better understanding of how
collaborative and competitive video games can be used to
teach programming (and other topics). There are a vast
number of ways that a game can be designed to teach com-
puter programming. Space Race is only one example of a
collaborative and competitive video game that can be incor-
porated into a course. Further investigation is necessary to
fully judge the educational effectiveness. Specifically, this
study was able to show, in a limited manner, that knowledge
acquired from gameplay can be retained by some students
over 7 weeks. This was shown with three questions on the
final exam. Future work should include a greater amount of
data, potentially over a longer time period, so as to judge
knowledge retention more completely.
Given the relative success of Space Race, the question
posted at the outset of this paper can be answered in the
affirmative: a competitive and collaborative video game
can be used to improve learning when teaching computing
to post-secondary students. The work presented here will
hopefully motivate other educators and researchers to use
collaborative and competitive video games to motivate and
engage this generation’s learners.
Acknowledgments The support from the Faculty of Engineering,
McMaster University, is gratefully acknowledged, as is the participa-
tion of the first year engineering students. The specific individuals
that we would like to thank for their contributions are Christopher
Anand, Kevin Browne, Andrew Curtis, Douglas Down, Steve Drekic,
and Michael Viveros.
References
Andersen E, O’Rourke E, Liu Y-E, Snider R, Lowdermilk J, Truong
D, Cooper S, Popovic Z (2012) The impact of tutorials on games
of varying complexity. In: Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems. ACM, pp 59–68
Annetta LA, Minogue J, Holmes SY, Cheng M-T (2009) Investigating
the impact of video games on high school students’ engagement
and learning about genetics. Comput Educ 53(1):74–85
Baloian N, Pino JA, Vargas R (2013) Tablet gestures as a
motivating factor for learning. In: Proceedings of the 2013
Chilean Conference on Human - Computer Interaction, ChileCHI
’13, pp 98–103,, NY, USA. ACM. ISBN 978-1-4503-2200-3.
doi:10.1145/2535597.2535622
Barab S, Dede C (2007) Games and immersive participatory sim-
ulations for science education: an emerging type of curricula.
J Sci Educ Technol
Journal of Science Education and Technology 16(1):1–3, 2. ISSN
1059-0145. doi:10.1007/s10956-007-9043-9
Barnes T, Powell E, Chaffin A, Lipford H (2008) Game2Learn:
improving the motivation of CS1 students. In: Proceedings of the
3rd International Conference on Game Development in Computer
Science Education, GDCSE ’08, pp 1–5, New York, NY, USA.
ACM. ISBN 978-1-60558-057-9. doi:10.1145/1463673.1463674
Bierre K, Ventura P, Phelps A, Egert C (2006) Motivating OOP by
blowing things up: an exercise in cooperation and competition in
an introductory java programming course. ACM SIGCSE Bulletin
38(1):354–358
Bland JM, Altman DG (1995) Multiple significance tests: the Bon-
ferroni method. BMJ?[Br Med J] 310(6973):170–170, 01. http://
www.ncbi.nlm.nih.gov/pmc/articles/PMC2548561/
Boyce A, Doran K, Campbell A, Pickford S, Culler D, Barnes T (2011)
Beadloom game: adding competitive, user generated, and social
features to increase motivation. In: Proceedings of the 6th Interna-
tional Conference on Foundations of Digital Games, FDG ’11, pp
139–146, New York, NY, USA. ACM. ISBN 978-1-4503-0804-5.
doi:10.1145/2159365.2159384
Browne K, Anand C (2013) Gamification and serious game approach
for introductory computer science tablet software. In: Gamifica-
tion 2013. University of Waterloo, Stratford Ontario
Burton-Chellew MN, Ross-Gillespie A, West SA (2010) Cooperation
in humans: competition between groups and proximate emotions.
Evol Hum Behav 31(2):104–108
Chaffin A, Doran K, Hicks D, Barnes T (2009) Experimental evalua-
tion of teaching recursion in a video game. In: Proceedings of the
2009 ACM SIGGRAPH Symposium on Video Games, Sandbox
’09, pp 79–86, New York, NY, USA. ACM. ISBN 978-1-60558-
514-7. doi:10.1145/1581073.1581086
Chan S (2014) The educational effectiveness of a cooperative and
competitive video game for teaching introductory programming.
Master’s thesis, McMaster University, Hamilton, ON, Canada
Chan S, Smith WS (2014) A cooperative competitive video game for
teaching introductory computer programming. In: ICEER 2014,
p4
Cheng K-H, Tsai C-C (2013) Affordances of augmented reality in sci-
ence learning: suggestions for future research. J Sci Educ Technol
22(4):449–462
Cockburn A, Bryant A (1998) Cleogo: collaborative and multi-
metaphor programming for kids. In: Proceedings of the Computer
Human Interaction, 1998. 3rd Asia Pacific. IEEE, pp 189–194
Connolly TM, Stansfield M, Hainey T (2007) An application of
games-based learning within software engineering. Br J Educ
Technol 38(3):416–428
Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM (2012)
A systematic literature review of empirical evidence on computer
games and serious games. Comput Educ 59(2):661–686
Dunleavy M, Dede C, Mitchell R (2009) Affordances and limita-
tions of immersive participatory augmented reality simulations
for teaching and learning. J Sci Educ Technol 18(1):7–22. ISSN
1059-0145. doi:10.1007/s10956-008-9119-1
Eagle M, Barnes T (2008) Wu’s castle: teaching arrays and loops
in a game. SIGCSE Bull 40(3):245–249. ISSN 0097-8418.
doi:10.1145/1597849.1384337
Ebner M, Holzinger A (2007) Successful implementation of user-
centered game based learning in higher education: an example
from civil engineering. Comput Educ 49(3):873–890. ISSN 0360-
1315. doi:10.1016/j.compedu.2005.11.026
Ford Jr CW, Minsker S (2003) TREEZ—an educational data structures
game. J Comput Sci Coll 18(6):180–185
Hicks A (2010) Towards social gaming methods for improving game-
based computer science education. In: Proceedings of the Fifth
International Conference on the Foundations of Digital Games,
FDG ’10, pp 259–261, New York, NY, USA. ACM. ISBN 978-1-
60558-937-4. doi:10.1145/1822348.1822386
Kahn K (1999) A computer game to teach programmming. In:
National Educational Computing Conference: NECC ’00: Spot-
light on the Future: Technology for the New Millenium
Kazimoglu C, Kiernan M, Bacon L, Mackinnon L (2012) A seri-
ous game for developing computational thinking and learning
introductory computer programming. Procedia Soc Behav Sci
47:1991–1999
Kelleher C, Cosgrove D, Culyba D, Forlines C, Pratt J, Pausch
R (2002) Alice2: programming without syntax errors. In: User
Interface Software and Technology. Citeseer
Kruskal WH, Allen WW (1952) Use of ranks in one-criterion variance
analysis. J Am Stat Assoc 47(260):583–621
Kumar B, Khurana P (2012) Gamification in education-learn computer
programming with fun. Int J Comput Distributed Syst 2:46–53
Li F, Zhao J, Shih TK, Lau R, Li Q, McLeod D (2008) Introduc-
tory C programming language learning with game-based dig-
ital learning. In: Advances in Web Based Learning - ICWL
2008, volume 5145 of Lecture Notes in Computer Science, pp
221–231. Springer Berlin Heidelberg. ISBN 978-3-540-85032-8.
doi:10.1007/978-3-540-85033-5 22
Long J (2007) Just for fun: using programming games in software pro-
gramming training and education—a field study of IBM robocode
community. J Inf Technol Educ 6:279–290
Maloney J, Burd L, Kafai Y, Rusk N, Silverman B, Resnick M (2004)
Scratch: a sneak preview. In: Proceedings of the Second Inter-
national Conference on Creating, Connecting and Collaborating
Through Computing, C5 ’04. IEEE Computer Society, Washing-
ton, pp 104–109. ISBN 0-7695-2166-5. doi:10.1109/C5.2004.33
Manos MVGKK (2005) Combining software games with education:
evaluation of its educational effectiveness. Educ Technol Soc
8:54–65. http://ifets.info/journals/8 2/ets 8 2.pdf#page=59
Martin A (2000) The design and evolution of a simulation/game
for teaching information systems development. Simul Games
31(4):445–463
McNemar Q (1947) Note on the sampling error of the difference
between correlated proportions or percentages. Psychometrika
12(2):153–157
Moore DS (1976) Chi-square tests. Defense Technical Information
Center
Muratet M, Torguet P, Jessel J-P, Viallet F (2009) Towards a serious
game to help students learn computer programming. Int J Comput
Games Tech 2009:3
Navarro EO, van der Hoek A (2004) Simse: an educational simula-
tion game for teaching the software engineering process. ACM
SIGCSE Bulletin 36(3):233–233
Nickel A, Barnes T (2010) Games for CS education: computer-
supported collaborative learning and multiplayer games. In: Pro-
ceedings of the Fifth International Conference on the Foundations
of Digital Games, FDG ’10, pp 274–276, New York, NY, USA.
ACM. ISBN 978-1-60558-937-4. doi:10.1145/1822348.1822391
Opie C, Sikes PJ (2004) Doing Educational Research. SAGE Pub-
lications. ISBN 9780761970026. https://books.google.ca/books?
id=XjBJltyWTzAC
Palincsar SA (1998) Social constructivist perspectives on teaching and
learning. Annu Rev Psychol 49(1):345–375. doi:10.1146/annurev.
psych.49.1.345
Papastergiou M (2009) Digital game-based learning in high school
computer science education: impact on educational effective-
ness and student motivation. Computers & Education 52(1):1–
12. ISSN 0360-1315. doi:10.1016/j.compedu.2008.06.004. http://
www.sciencedirect.com/science/article/pii/S0360131508000845

Perkins DN, Simmons R (1988) Patterns of misunderstanding: an inte-
grative model for science, math, and programming. Rev Educ Res
58(3):303–326
Perneger TV (1998) What’s wrong with Bonferroni adjustments.
BMJ?[Br Med J] 316(7139):1236–1238, 04. http://www.ncbi.nlm.
nih.gov/pmc/articles/PMC1112991/
Pirker J, Riffnaller-Schiefer M, Gütl C (2014) Motivational active
learning: engaging university students in computer science edu-
cation. In: Proceedings of the 2014 Conference on Innovation
& Technology in Computer Science Education, ITiCSE ’14, pp
297–302, New York, NY, USA. ACM. ISBN 978-1-4503-2833-3.
doi:10.1145/2591708.2591750
Prensky M (2007) Digital game-based learning. Paragon House
Probst TM, Carnevale PJ, Triandis HC (1999) Cultural values in inter-
group and single-group social dilemmas. Organ Behav Hum Decis
Process 77(3):171–191
Rajaravivarma R (2005) A games-based approach for teach-
ing the introductory programming course. SIGCSE Bull
37(4):98–102. ISSN 0097-8418. doi:10.1145/1113847.1113
886
Sampaio B, Morgado C, Barbosa F (2013) Building collaborative
quizzes. In: Proceedings of the 13th Koli Calling International
Conference on Computing Education Research, Koli Calling ’13,
J Sci Educ Technol
pp 153–159, New York, NY, USA. ACM. ISBN 978-1-4503-2482-
3. doi:10.1145/2526968.2526985
SciPy Developers (2014) Scipy.org. http://www.scipy.org/, 2014
Accessed: July 19
Smith H (2014) Spaceteam. http://www.sleepingbeastgames.com/
spaceteam/, July 2014. http://www.sleepingbeastgames.com/
spaceteam/. Accessed: July 2
Tinto V (2000) Taking student retention seriously: rethinking the first
year of college. NACADA J 19(2):5–10. http://suedweb.syr.edu/
Faculty/Vtinto/Files/AACRAOSpeech.pdf
Vorderer P, Hartmann T, Klimmt C (2003) Explaining the enjoyment
of playing video games: the role of competition. In: Proceedings of
the second international conference on Entertainment computing,
pp 1–9. Carnegie Mellon University
Wendel V, Gutjahr M, Göbel S, Steinmetz R (2013) Design-
ing collaborative multiplayer serious games. Education and
Information Technologies 18(2):287–308. ISSN 1360-2357.
doi:10.1007/s10639-012-9244-6
Wouters P, Nimwegen CV, Oostendorp HV, Van Der Spek ED (2013)
A meta-analysis of the cognitive and motivational effects of
serious games. J Educ Psychol 105(2):249
Yoon EKS (2005) Developing games and simulations for today and
tomorrow’s tech savvy youth. TechTrends 49:33–41