Conservation of Mass: Results of Usability/Alpha Testing

Subject Matter Expert (SME) Survey Results

Both are Florida Virtual School (FLVS) teachers have worked in the classroom and
virtually as science teachers. They both have 10+ years of science teaching experience
at the high school level.

SME Usability Survey

1. The objectives for Strongly Disagree Neither Agree Strongly
the Battle for Florida Disagree Agree or Agree
City computer game Disagree
ranged in a variety of
Blooms Taxonomy
levels.

SME 1 X
SME 2 X
2. The objectives of Strongly Disagree Neither Agree Strongly
the Battle for Florida Disagree Agree or Agree
City computer game Disagree
were aligned with the 3
different
challenges/activities.

SME 1 X
SME 2 X
3. The multiple- Strongly Disagree Neither Agree Strongly
choice questions to Disagree Agree or Agree
enter each level of the Disagree
computer game
represent various
knowledge levels of
physical and chemical
changes that occur in
reactions.

SME 1 X
SME 2 X
4. The Battle for Strongly Disagree Neither Agree Strongly
Florida City computer Disagree Agree or Agree
game instructions were Disagree
simple to follow and
clearly stated.

SME 1 X
SME 2 X
5. The Battle for Strongly Disagree Neither Agree Strongly
Florida City computer Disagree Agree or Agree
game instructions were Disagree
simple to follow and
clearly stated.

SME 1 X
SME 2 X
6. What activity did you enjoy the most about the Battle for Florida City computer
game?
SME 1 I liked that the questions got harder as we went (also the
superhero theme) and that there is the ability to review if
you got a question wrong.
SME 2 I enjoyed the last activity the most as it required more
thought to balance the equations, so was more challenging
for students.
7. What would you change about any of the activities in the Battle for Florida City
computer game?
SME 1 On floor 4, after getting the answer correct - in the
feedback, show the balanced equation with coefficients for
the real reactions to show what they would look like. Well
done!!! This was fun!
SME 2 I liked the activity, but would give directions letting them
know to enter the elevators you need to mouse over them.
It was a bit frustrating not know this. In draining the liquid
N2 tank for the second activity with the numbers. Would it
be better to have one of the product numbers fixed for the
first problem? On the 3rd floor it said, "You have located
the first container of liquid N2" but I had already done floor
2. The last floor activity seemed too long as it kept
repeating similar problems. Maybe cut to one of each
type? Overall very fun and engaging. :)
One-on-One Evaluation with Students
The 3 test students were varied from grade 6th-9th and are all currently enrolled in a
science course at school. All of them are very accustomed to completely tasks online,
either for school or in their free time.

One-on-One Evaluation with Students

1. I enjoyed playing Strongly Disagree Neither Agree Strongly
Battle for Florida Disagree Agree or Agree
City Disagree

Student 1 X
Student 2 X
Student 3 X
2. I found the Strongly Disagree Neither Agree Strongly
screen layout easy to Disagree Agree or Agree
use Disagree

X
Student 2 X
Student 3 X
3. I became Strongly Disagree Neither Agree Strongly
frustrated while playing Disagree Agree or Agree
the game. Disagree

Student 1 X
Student 2 X
Student 3 X
4. I understood the Strongly Disagree Neither Agree Strongly
story and what I was Disagree Agree or Agree
trying to achieve while Disagree
playing the game.

Student 1 X
Student 2 X
Student 3 X
5. The graphics Strongly Disagree Neither Agree Strongly
grabbed my attention. Disagree Agree or Agree
Disagree
Student 1 X
Student 2 X
Student 3 X
6. The graphics Strongly Disagree Neither Agree Strongly
made me want to play Disagree Agree or Agree
the game. Disagree
Student 1 X
Student 2 X
Student 3 X
7. The game Strongly Disagree Neither Agree Strongly
instructions were easy Disagree Agree or Agree
to follow. Disagree

Student 1 X
Student 2 X
Student 3 X
8. The text was Strongly Disagree Neither Agree Strongly
presented in a clear Disagree Agree or Agree
and easy to read Disagree
manner

Student 1 X
Student 2 X
Student 3 X
9. The difficulty Strongly Disagree Neither Agree Strongly
was appropriate for Disagree Agree or Agree
me. Disagree

Student 1 X
Student 2 X
Student 3 X
10. I stayed focused Strongly Disagree Neither Agree Strongly
and interested in Disagree Agree or Agree
playing the game. Disagree

Student 1 X
Student 2 X
Student 3 X
11. Challenge 1 Strongly Disagree Neither Agree Strongly
was easy to complete. Disagree Agree or Agree
Disagree
Student 1 X
Student 2 X
Student 3 X
12. I learned Strongly Disagree Neither Agree Strongly
something new when Disagree Agree or Agree
completing challenge Disagree
1.
Student 1 X
Student 2 X
Student 3 X
13. What I liked about Challenge 1 was...
Student 1 No response
Student 2 It was a simple and easy to use layout, paired with a slightly
challenging puzzle made for an enjoyable experience.
Student 3 Interaction mechanics
14. What I would recommend changing about Challenge 1 is....
Student 1 No response
Student 2 I really wouldn't change anything. It fits it well.
Student 3 I had a hard time connecting the challenge to the concept.
15. Challenge 2 Strongly Disagree Neither Agree Strongly
was easy to complete. Disagree Agree or Agree
Disagree
Student 1 X
Student 2 X
Student 3 X
16. I learned Strongly Disagree Neither Agree Strongly
something new when Disagree Agree or Agree
completing challenge Disagree
2.

Student 1 X
Student 2 X
Student 3 X
17. What I liked about Challenge 2 was...
Student 1 No response
Student 2 A change! I was not expecting this to be as challenging as it
was. I enjoyed this one more than the first one, for sure.
Student 3 Different interaction mechanics. Variety in gameplay is good.
18. What I would recommend changing about Challenge 2 is....
Student 1 No response
Student 2 Not really anything can be changed. It looks nice and feels
nice.
Student 3 I wasnt clear on the connection of the challenge to the actual
problem.
19. Challenge 3 Strongly Disagree Neither Agree Strongly
was easy to complete. Disagree Agree or Agree
Disagree
Student 1 X
Student 2 X
Student 3 X
20. I learned Strongly Disagree Neither Agree Strongly
something new when Disagree Agree or Agree
completing challenge Disagree
3.

Student 1 X
Student 2 X
 Student 3 X
21. What I liked about Challenge 3 was...
Student 1 No response
Student 2 It was hard! We have barely covered the topic of the last
puzzle in the section in my science class, so this was new
and challenging!
Student 3 Logical extension of previous exercise. I like the idea of
progressively refining abstractions to get to concepts.
22. What I would recommend changing about Challenge 3 is....

Student 1 No response
Student 2 Once again, this game is very nice, especially considering
that it is only in Alpha! I wouldn't change much at all, really.
Student 3 The game interface is a little cluttered.
23. The game Strongly Disagree Neither Agree Strongly
helped me learn about Disagree Agree or Agree
the law of conservation Disagree
of mass.

Student 1 X
Student 2 X
Student 3 X
24. The game Strongly Disagree Neither Agree Strongly
helped me learn about Disagree Agree or Agree
balancing equations. Disagree
Student 1 X
Student 2 X
Student 3 X
25. Three things that I learned while playing the game.
Student 1 Understanding the concept visually
Student 2 1. I learned about different elements and molecules and
how they fit together.
2. I learned about the law of conservation.
3. I had a refresher on chemical and physical changes.
Student 3 It may have been more basic than my actual curriculum.

Summary of Results:

After completing student and SME usability surveys, the game meets the objectives as
intended. According to the SMEs, each challenge follows the Law of Conservation of
Mass and increases in inquiry based learning throughout. There are some
improvements or tweaks to be made, however very minor.
Overall, the student testers found the game enjoyable and fun, and commented on
enjoying the variety of progressively difficult challenges. They also reported improving
their knowledge or building on previous knowledge on the law of conservation of mass.

Improvements to be Made:

Edit directions
Entering elevators
Clarify for N2 tank
Eliminate some problems in the last challenge
Simplify game interface; remove any unnecessary visual clutter.