Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Brandy Milson
Formative Evaluation
Dr. Rice
November 2, 2010
Running head: FORMATIVE EVALUATION 2
There are many educational tools and programs available to educators regardless of the level
of instruction needed and the readiness level of students. When asked to choose one to evaluate,
the choices can be overwhelming. "A formative evaluation is the process of evaluating the
instructional product during its design and development. The purpose of formative evaluation is
to review the instruction for weaknesses and make the necessary revisions to correct errors and
enhance the effectiveness before implementation" (Gayle & Rasmussen, 2006, p. 133).
Formative evaluations are often done by expert reviewers and a sample of the target audience
(tryout) such as a selection of students in your classroom. For this formative evaluation, I have
chosen to analyze and critique is Shopping Spree offered by e-Learning for Kids and it can be
designed to provide age 12 math students a lesson on calculating the percentage of a number.
According to Gayle & Rasmussen (2006), "Evaluation is the means for determining the value,
or worth of an instructional product or program" (p. 128). Therefore, the goal of this formative
instructional effectiveness, efficiency, and appeal" (Gayle & Rasmussen, 2006, p. 128). Gayle
and Rasmussen argue that effectiveness, efficiency and appeal are highly influential on learner
Effectiveness
Gayle and Rasmussen (2006) contend that," effectiveness of WBI is measured by determining
whether learners achieved the instructional goals (p. 128)". The effectiveness criterion evaluates
Running head: FORMATIVE EVALUATION 3
the mastery of goals and success of instruction. The instructional goal of Shopping Spree is for
students to learn how to calculate the percent of a number. Content is introduced through an
interactive visual and audio lesson. After each concept is introduced you have to calculate a
percentage which earns you money to spend on a Shopping Spree. There are five venues
(restaurant, clothing store, bookstore, sports store, and shoe store) for practicing calculating
percentages (as sales prices). I played the game several times and there are different shopping
Efficiency
Where there is a forced choice or tradeoff required for time, monies and other resources
efficiency is the quality that you must examine closely. When examining the efficiency of a
program, stakeholders primarily look at the financial benefits as opposed to the costs. Does the
educational tool or program provide a savings of either time, money, or other resources. Is the
instruction delivered in a manner that saves money and time for stakeholders? Shopping Spree is
a free educational program offered by e-Learning for Kids which is a private non-profit
organization for kids . For many primary stakeholders, there is a savings of time and money.
Appeal
"Appeal involves gaining and maintaining learner's attention on and interest in the
instructional task. Appeal usually focuses on the motivational aspects of instructional content
and WBI features (activities, message design, etc.)" (Gayle & Rasmussen, 2006, p. 128).
Appeal is evaluated mainly by its ability to gain and maintain learner interest and the usability of
the overall design and navigational elements. Shopping Spree has appeal for learners as it places
them in real life situations of having to figure out sales prices or tips for restaurant servers and
in a colorful and storytelling manner. The lesson is presented by the Math Wizard who floats on
Running head: FORMATIVE EVALUATION 4
the screen and explains through text and sound the ways that users can calculate percentages.
Throughout instruction the wizard asks them to perform simple calculations. After answers are
entered the wizard shows the steps to working out the problem and provides a detailed
explanation of the solution. Then as you practice the method the Math Wizard just taught you
make a basket into the basketball net each time you get an answer right or you pop a balloon on a
carnival board. You get immediate effect that is reinforcing and entertaining.
To conduct a formative evaluation there are several steps which must be taken to ensure a
comprehensive evaluation. The first thing that an evaluator needs to do when constructing a
Instructional Goals
This evaluative area assesses the accuracy, clarity, completeness and congruence of
instructional goals and objectives. (Gayle & Rasmussen, 2006, p.136) In other words, for the
instructional program Shopping Spree by e-Learning for Kids, users are asked to learn how to
calculate percentages. An evaluator would check to see if the content is accurate, determine if
the instructional goal and objective of calculating percentages was clear for users, decide if the
examples and information covered is sufficient for mastering objectives, and finally they would
check to see if all exercises and games associated with the program match objectives and goals.
Instructional Content
After a program or tools instructional goals have been analyzed, the next step is to look closely
at the instructional content. First, determine if it is accurate and presented in the appropriate
sequence. In the case of Shopping Spree, all information was accurate and the assessment
questions and games where presented after the instructional information which is a logical
Running head: FORMATIVE EVALUATION 5
sequence for most learners. Next, an evaluator should check practice exercises and directions for
clarity and completeness to ensure that learners working on their own are not confused or
frustrated. Once completeness and accuracy has been checked, it is important to determine the
amount and quality of instructional detail. Is it enough to support the concepts or skills being
taught. In the case of Shopping Spree, there is sufficient level of content if it is meant to be used
as a review or enrichment exercise but there is not enough detail if it is meant to be a standalone
instructional unit. Finally, a designer has to ask if the instructional content is appealing. It
doesn’t matter how effective your instructional program is, if it is not appealing and will not
Technology
Once instructional goals and content are evaluated you must examine the technological
elements of the program or instructional tool. According to Gayle and Rasmussen (2006),
punctuation, and word usage" (p.136). Once the functional elements are tested it is time to
analyze technology applications such as hypermedia links. You have to examine all of the
website elements and makes sure that they are functioning properly and are instinctive. Finally,
when looking at technological aspects it is important to assess the availability of support and
assistance from the product which helps prevent or solve technical problems that may arise.
When looking at Shopping Spree I located no functionality errors and there are no hypermedia
links just links within the instructional program which all is functioning properly. The one
major downfall I found was a lack of instructional support but the company does give an address,
email, and website where questions and concerns can be addressed. Depending on who
Message Design
Once all other elements are in place it is time to analyze the message design of the program or
instructional tool. Is the media pleasing? Does it fit with an overall theme? What are the
graphics like? Are they emphasizing, distracting, or entertaining? Are graphical visual and
audio elements working properly and do they present a cohesive message? Are the navigational
elements clear and relevant to goals and navigation? When examining the message design of
Shopping Spree, I found that it was well designed and the overall theme and instructional goal
was clear throughout the instructional lesson and practice exercises. The graphics were visually
pleasing and fit within the overall shopping theme and they were both emphasizing and
entertaining but were not distracting at all. All navigational elements are consistently located in
a bottom bar with unified icons and identifying text rollovers in case there was any confusion.
Identifying Stakeholders
The first step is to identify the Stakeholders are who evaluative information is reported to.
There are two types of stakeholders which include primary and secondary stakeholders. Primary
stakeholders are more directly involved in production and delivery. Often, they have ultimate
decision making authority. For Shopping Spree the Primary Stakeholders are the global
nonprofit organization called e-Learning for Kids or EFK. They designed the program and are
the only ones with editing or implementation privileges. The learners that use the program are
also primary stakeholders as they provide feedback (or their parents do) on the success of the
program. There are also some secondary stakeholders who are indirectly affected or loosely
responsible for or interested in the instructional program or tool. In Shopping Spree there could
be a number of secondary stakeholders such as Curriculum and Instruction specialist that are
Running head: FORMATIVE EVALUATION 7
using the program in their schools with students, parents, teachers, or possibly even school
boards.
Evaluative Product
Once the stakeholders have been identified it is time to determine the evaluative product(s).
"What is being evaluated?, focuses on the specific instructional materials, processes, or products
that will be examined during formative evaluation" (Gayle & Rasmussen, 2006, p. 144). There
are many types of design materials such as "design plans, storyboards, instructional content,
instructional strategies, interface, navigational features, message and visual design features,
WBI prototypes, assessment tools and scores, practice exercises and more" (Gayle & Rasmussen,
storyboards, design plans, and other preplanning manipulatives because that information is often
not made public. Shopping spree was analyzed using the interface which is easy to get around
and interpret, the navigational features which are clearly located in an uniform location
throughout the lesson or practice exercises, the message and visual design which were of high
quality, and the practice exercises which fit perfectly with the theme and were an accurate
The next step in the formative evaluation process is to identify your evaluations and
reviewers. According to the Joint Committee on Standards for Educational Evaluation, "An
evaluator should be someone who is competent and trustworthy, who possesses substantive
knowledge, technical competence, integrity, experience, public relation skills, and other
the purposes of this assignment, I am the reviewer and evaluator. I have a number of the
Running head: FORMATIVE EVALUATION 8
recommended requirements set forth by Gayle and Rasmussen. Once an evaluator is established,
it is important that their roles are based in a purpose and the tasks requirement to fulfill that
purpose. The main purpose of the evaluator is to determine strengths and weaknesses of the
evaluation. In shopping Spree there are a lot of educational strengths, many of which I have
already discussed, but there are also a few weaknesses which must be revised. I think that there
should be levels of difficulty which can increase with user correct responses. This would allow
for some growth with the game which will increase use. I also think that it would be more
appealing if there were more locations added to the menu so that there would be an increase real
world effectiveness of practice which will increase knowledge retention. The final element to
coursework. I also have a good in-depth understanding of calculating percentages which is the
Evaluation Methods
evaluation which best fit the instructional product and need. One must determine how they are
going to gather data. The method is usually determined by the program or tools themselves or the
evaluative purpose itself. "Practical matters such as time, budget, and logistics influence the
selection of methods and tools" (Gayle & Rasmussen, 2006, p. 147). Some of the more common
methods include surveys, observations, pre or post testing, expert or end user reviews, and extant
data such as organizational reports, emails and discussions. (Gayle & Rasmussen, 2006, p. 147)
The best thing to keep in mind when selecting a method is to keep it simple. For a product like
Shopping Spree, it would most likely be best to gather data through observations, surveys, or
Running head: FORMATIVE EVALUATION 9
other factors which can be located online or created locally. Since instruction will take place
One thing that distinguishes WBID evaluations and traditional formative evaluations is that
they are ongoing until initial implementation. There is no traditional waiting period and as
mistakes are found they are rapidly corrected. The first evaluation occurs on the design plans.
"The design team, experts, and/or end users evaluate the WBI design plan for the clarity of its
goals, meaningful examples, and relevant instructional activities. They may review the
instructional and motivational strategies for their effectiveness, appropriateness, and potential
learner motivation" (Gayle & Rasmussen, 2006, p. 153). Although traditional models allow for
one to one, small group and field test tryouts WBID often relies on some form of tryout with end
users. An evaluator can create their own sample population to tryout material with or often the
initial implementation is the tryout for the program. It is impossible to determine the Shopping
Spree evaluation location and timing without access to their pre-planning material and design
documents.
Decisions
Once evaluations have been completed they need to be analyzed which can be accomplished
nonparametric tests such as a chi-square. You can also use qualitative analyzational tools such
as an interview with end users. Once data has been analyzed, modifications and
Communicating Results
Forms of Communication
There are two ways of communicating evaluation processes and results. Verbal reports are
easy to share and can be done quickly if immediate access to stakeholders is possible. If I
wanted to report my findings to EFK, I would have to generate a written report which is more
formal and permanent. If the option is available a combination or written and verbal reports is
the best strategy but this is not often the case. Regardless of the form consistency is important.
"Other critical features of reporting practices include timeliness, brevity, clarity, and
responsiveness to the audience" (Gayle & Rasmussen, 2006, p. 159). Evaluators must always
keep stakeholders in mind when communicating evaluation results but be careful not to engage
in communication overload.
Verbal reports. These are primarily used to report project progress, problems, and technical
issues. You can deliver verbal messages on the phone, face to face or through online webcasting
or communication. It is important to remember these types of reports are temporary. If you want
anything permanent or set it in stone it is important to document as much as possible and get
verbal reports were made on Shopping Spree without access to formative documents.
Written reports. These reports can serve as progress or final reports. They can take the
form of memos, letters, emails, or other forms of written communication. Some designers and
evaluators use templates which make the communication process easier. It is also impossible to
report on the written reports generated for Shopping Spree without inside access to the
Final Report. The final report is often presented as an executive summary with full
documentation. This report discusses evaluation activities, findings, and recommendations for
stakeholders. Once you have your final report you can now publish your executive summary
User Profile: Elementary and middle school students studying percentages in math class
documents
Are the goals and Examine This is often The implied goals
objectives of the goals and reported to within Shopping
educational tool or objectives students is Spree are clear and
program clear? for clarity. lessons narrowly focused on
Think about overviews and calculating
smart goals on course percentages.
syllabi
Are the goals or Think about Reported to If goals are not
objectives of the SMART students achievable then they
educational tool or goals? through are incomplete. You
program achievable? assignments can achieve an
and on the understanding of
course syllabi how to compute
percentages through
Shopping Spree.
Are the goals and Does the This is often Online delivery is the
content appropriate for delivery not reported to only way that you
the method of delivery? system students can deliver the
match the Shopping Spree
content program
What educational need(s) Examine the The Shopping Spree
does the content address? educational educational focuses on the need
need closely. need is for students to apply
If there is no outlines in the percentage
need there is syllabus and calculations to real
no need to scope and life scenarios
use the sequence
program or documents
tool
Are the educational Examine the Objectives will Shopping Spree does
objectives clearly stated? product or be stated not explicitly state
Content
using
Shopping
Spree.
Is the content appropriate Determine This is often Shopping spree is
to the stated audience? appropriate not reported to designed for
age range students. elementary and
middle school math
students
appropriately.
Is the information Make sure If any Shopping Spree
complete, covering the there are no omissions are seems complete.
content properly? accidental discovered you
omissions can report to
for content students
verbally or
through written
directive.
Are reference citations Check for a Report Shopping Spree has
provided? references references to no reference
page students citations.
through written
directive so
that they may
look up
primary
sources if they
need to
Do the instructional Look at You can report Shopping Spree is
activities promote Blooms to students more of a drill and
thoughtful and reflective taxonomy. verbally or practice program.
responses or Determine through written
discussions? the level of directive.
instruction
Do the activities Look at the You can report The Shopping Spree
promote learning? activities to to students activities promote
determine if verbally or learning.
they are through written
educational directive.
in nature.
Does the site contain any Look closely You wouldn’t This is very
extraneous and
for possible report this to important to
unsuitable vocabulary,
language or concepts, inappropriate students if stakeholders and
bias, or stereotyping? language, these elements there are no loaded
bias, and exist, you language present in
stereotyping. simply Shopping Spree.
wouldn’t use
Running head: FORMATIVE EVALUATION 15
the program or
tool.
Does the program project Look for You can report There were no biases
any obvious bias? Are bias and to students or misleading
there an errors or errors verbally or omissions in
misleading omissions? through written Shopping Spree,
directive.
Does the content Analyze the You can report Shopping Spree
emphasize depth or primary to students emphasized
repetition? instructional verbally or repetition
task through written
directive.
Is the content subject Look for a You would not Shopping spree had
matter interesting, lively, story, look report this to an excellent story
or compelling? for elements students but line and was
of interest they will interesting and
surely report engaging.
this to you
Is there a difficulty range Look for a You can report Shopping spree did
available? difficulty to students not have a difficulty
range for verbally or range but it was
enrichment through written designed for age 12.
or recurring directive.
use
Look for You can report
Are references,
supporting to students
bibliographies, or other
material through written
supporting evidence
directive.
provided?
trip.
Is there quality student Is there Is there quality Is there quality student
interaction with content? quality student interaction with
student interaction with content?
interaction content?
with content?
Once students have Test this with You can report Shopping spree is easy
learned the basics of the students and to students to use and has on-
software is it easy to use? aids verbally or going instruction
through written prompts
directive.
Do students require Determine the You can report Shopping Spree
additional training amount of to students requires no training.
training verbally or
needed to use through written
the program directive.
or tool
Running head: FORMATIVE EVALUATION 17
Is the program easy to Test this with You can report Shopping Spree is very
learn to use for first time students and to students simple to use on the
students? Have students aids verbally or first time because of
learned how to use similar through written on-going directions.
programs? directive. First time use is
important because end
users will never come
back if they are
confused or frustrated
by their initial
experience
Do supporting graphics Examine You can report Shopping Spree has
and features enhance each graphic to students excellent enhancing
learning without for useful verbally or graphics that
distracting? ness and through written illustrate the
purpose directive. Most concepts well.
often students
will report it to
you
Is the appropriate voice Analyze the You can report Shopping Spree uses
used in expressing the tone of to students a Math Wizard
content to the learners? voice and verbally or character that is
determine if through written appropriately
it is the right directive. Most matched to the target
fit often students age group
will report it to
you
Was humor used Look for You can report Shopping Spree
appropriately? uses of to students doesn’t really use
humor and verbally or any humor.
determine if through written
appropriate/ directive. Most
inappropriate often students
will report it to
you.
Are directions clear? Examine You can report Shopping Spree
directions to students directions are clearly
throughout verbally or and concisely
the entire through written presented.
educational directive. Most
program for often students
clarity will report it to
you.
Does the text stand- Look for You often do Shopping spree
alone if graphics are cross not report this doesn’t have much
unavailable? compatibility to students. stand-alone text.
and analyze
Running head: FORMATIVE EVALUATION 18
graphical
elements
Are the goals stated Examine This is often
clearly and concisely? goals and reported to
objectives students is
for clarity. lessons
Think about overviews and
smart goals on course
syllabi
Is there congruence Check for This is often Shopping Spree is
between the unity and mapped out for unified.
instructional goals and correlation students
content? through an
assignment
Goals
breakdown in a
course syllabus
Is the content Examine if This is often Shopping Spree is
information clearly and information reported to clear and concise
Efficiency Questions
Are there synchronous Examine the You can report Shopping Spree
and asynchronous types types of to students doesn't use
of activities for students activities. verbally or synchronous and
and instructor? Remember through written asynchronous types
variety is directive. of activities
important
Are the goals relevant to Ask learners They usually Shopping Spree's
learners? what they inform you goals are relevant to
hope to get first. learners as they are
out of the often consumers of
instructional products which go on
program sale and they eat at
restaurants and must
Goals
Appeal (usability)
learn to calculate
tips.
Is the content Ask student The usually Shopping Spree is
interesting/challenging? aids and inform you. interesting and
others if the challenging.
program is
Content
interesting or
challenging
to them.
Running head: FORMATIVE EVALUATION 20
Does the software track Check to see You can Shopping Spree has
student progress? if there is a communicate no tracking tools
monitoring through verbal
tool which and written
tracks directives
student
progress and
performance
Does the program have Look for You can Shopping Spree has
speech capabilities accessibility communicate verbal and written
tools through through verbal text directives
the program and written written throughout
directives but no explicit
accessibility tools
*Does the site have the Ask student Students will Shopping Spree
potential for developing aids and often helps students gain
confidence and
satisfaction as a result of other is communicate confidence in their
learner effort? completion this to you. percentage
of the conversion skills.
program was
satisfying
and why or
why not?
Instructions for assessment: After testing all five located provided by the online educational program
Shopping Spree, evaluate the instructional program by circling the point Y or N in the boxes on the check
list. Evaluations will be collected and used for revisions.
Checklists
Program Evaluation
Quality Item Mandatory Possible Points
Y/N (only if the
Answer is yes
Are points marked
Shopping Spree includes clear directions for their instructional topic of choice Y 20
Shopping Spree has met the criteria for effectiveness, efficiency, and appeal Y 20
Multimedia elements of Shopping Spree are appealing and functional Y 10
Shopping Spree is age appropriate for its target audience Y 10
Shopping Spree provides appropriate feedback Y 10
Shopping Spree content is closely aligned with learning objectives Y 20
Shopping Spree is engaging and challenging for learners Y 10
Total Points 100
Final Grade
If a score of less than 60 points is obtained then instructional objectives have not been met and revisions must be
made to ensure quality
FeedbackComments:___________________________________________________________________________
_____________________________________________________________________________________________
_____________________________________________________________________________________________
_____________________________________________________________________________________________
Do you feel that that the instructional approaches used in this Shopping Spree met learner needs
and instructional objectives?-
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
Running head: FORMATIVE EVALUATION 26
Needed Actions
The chart above documents needed actions through several evaluation phases. Mostly it
is important to closely analyze individual elements and make revisions or clarify information as
needed. If you are using a program revised by others, often revisions are not possible. However,
you could clarify vague elements in students if they are small and do not interfere with the
Reporting to Students
There are many ways to report information to students. Often the most effective method is
simple written and verbal directives. When transmitting information to students it is important to
keep it simple and not cause cognitive overload or frustration. Sometimes reporting is as simple
Regardless of the method on some elements such as clarity, goals, and task difficulty
communication is important.
Conclusion
Formative evaluations are important to WBID because they assess the ongoing progress of a
program or tool. Programs must be analyzed for effectiveness, efficiency and appeal because
they are often predictors of successful programs and tools. You also must look at the four parts
to a formative plan and the six main steps involved in the design plan. Finally, you must
determine actions that you will take as well as how you will communicate evaluative
References
Branch, R., Kim, D., Koenecke, L. Evaluating Online Educational Materials for Use in
Education, Inc.
Herring, D. F., et. al., (2005). Multimedia software evaluation form for