Running head: Evaluation Proposal

Evaluation Proposal
Cristiane Vicentini
EME 640 Introduction to Program Evaluation
The University of Tampa
Spring 2015

Evaluation Proposal

2
Program Evaluation Proposal

Request for Proposal (RFP)

Global Enterprises, a Japanese electronic equipment manufacturing company with
headquarters in Tokyo, Japan, and a main branch in Sao Paulo, Brazil, has requested that an
instructional designer perform an evaluation of their newly developed English language learning
program: Distance Learning Solutions. This program was created as a means to provide further
business English language practice to Global Enterprises staff members in Sao Paulo, Brazil, in
order to allow them to better communicate with their counterparts at the headquarters in Japan.
Distance Learning Solutions was created as an asynchronous online program available at
http://distancelearningsolutions.weebly.com, with biweekly synchronous meetings held by an
instructor via Adobe Connect. It is still in its initial stages of implementation, and currently it
only allows staff members of intermediate or higher English language ability to participate in it.
Before an overall expansion of the program, which would include the creation of lessons
and material for beginner level learners, as well as the addition of more topic-based instructional
content to reach all staff members located in Sao Paulo, Brazil, Global Enterprises is requesting
proposals for its evaluation.
Overall Evaluation Plan
The evaluation plan of the Distance Learning Solutions language program (DLS) will
consist of three types of evaluation: formative, summative, and confirmative, each containing a
specific timeline and assessment items. Formative evaluation will be carried out first, with the
objective of assessing the level of success of DLS and determining what modifications need to
be made to ensure that the program reaches its highest potential. After an analysis of the results,

Evaluation Proposal

DLS will be modified accordingly. All the formative evaluation procedure will take place from
May to July of 2015.
The modified DLS language learning program then then be subject to a summative
evaluation, which will be conducted from August to October of 2015 to ensure that the newly
revised program is able to meet the needs of the learners who take part in it. After analyzing the
results of this evaluation, changes which might be deemed necessary will be made to ensure that
DLS is a successful tool to facilitate communication among Japanese and Brazilian Global
Enterprises staff members.
The final phase of the DLS program evaluation plan will consist of a confirmative
evaluation process, which will be carried out four months after the conclusion of the summative
evaluation. In February of 2016, evaluation tools will be used to assess the DLSs effectiveness
among staff members taking part in it and to provide further validation to this language program.
After it is confirmed that the program is successful in achieving its goals, the
instructional designer will present the results to the stakeholders at Global Enterprises, who can
then sanction the overall expansion of the program to reach Brazilian staff members of all
English language proficiency levels. The participants in the program and the stakeholders will
take part in a meeting where two $100.00 Amazon gift cards will be raffled among staff
members who participated in the entire program evaluation.
The data collection and assessment tools for each phase of the evaluation plan are
described next:

Evaluation Proposal

Formative Evaluation

Subject-Matter Expert Review

Evaluation of DLS website for content accuracy and technical quality
(Tessmer, 1993, p.47)
In-person interview with subject-matter expert to discuss possible changes

One-to-One Evaluation
Attitude and Acceptance survey (10 questions)
Interview of 5 participants in the DLS program
Interview of DLS program instructor

Summative Evaluation

One-to-One Evaluation
Interview of 10 participants in the DLS program
Interview of DLS program instructor
Attitude and Acceptance survey (10 questions)
English Language Proficiency post-test (20 questions)

Confirmative Evaluation

One-to-One Evaluation
Attitude and Acceptance survey (10 questions)
English Language Proficiency post-test (1 essay question)
Interview of DLS program instructor
At each stage of the program evaluation, there will be an interview with the DLS program

instructor, Ms. Cristiane Vicentini. She was responsible for the initial analysis of the placement
test results and the creation of the individual study guides for each participant in the program.

Evaluation Proposal

She also is responsible for providing feedback to learners regarding their speaking and writing
ability as they complete their tasks online. Therefore, she will provide valuable input in the
assessment of the improvement of learners participating in the program.
Following is a detailed explanation of each type of evaluation, which includes the
timeline for each stage, as well as sample assessment items.
Formative Evaluation
In the formative stage of the evaluation plan, there will be two types of evaluation: a
subject-matter expert review and a one-to-one evaluation. The following timeline will be
observed. Note: United States national holidays will be followed since the instructional designer
resides in the U.S.
TIMELINE FORMATIVE EVALUATION (May-July 2015)
May 1 May 20

May 21 May 30

May 31
June 1 June 30

July 1
July 2

Contact and recruit SME for evaluation

Develop evaluation tools (survey and
interviews)
Send out DLS information to hired SME
Schedule in-person interview with SME for
feedback and discussion of possible
changes (May 31)
Send interview questions to SME
Send out attitude and acceptance survey to
all participants in the program
Interview SME
Transcribe data
Make suggested changes to DLS website
Schedule interviews with 5 participants in
the program (July 1 July 8)
Schedule interview with DLS instructor
(July 9)
Interview Participant 1
Transcribe data
Interview Participant 2
Transcribe data

Evaluation Proposal
July 3
July 6
July 7
July 8
July 9

July 15
July 15 July 31

6
Interview Participant 3
Transcribe data
No interviews
Independence Day Holiday Observed
Interview Participant 4
Transcribe data
Interview Participant 5
Transcribe data
Send out reminder to take attitude and
acceptance survey to all participants in the
program
Interview DLS instructor
Transcribe data
Close attitude and acceptance survey
Analyze data from interviews and surveys
Make suggested changes to the DLS
website

Description of Procedure and Tools

First of all, a subject-matter expert (SME) from the United States will be hired to review
the DLS website for content accuracy and technical quality (Tessmer, 1993, p. 47).
Recruitment will take place among top consulting companies specialized in language-teaching
material. After the material has been evaluated by the SME, an in-person interview between the
instructional designer and the SME will take place in order to find out about the quality of
material, the accuracy of the content, and possible suggestions for improvement. The SME will
have 10 days to evaluate the DLS website. The SME will also be sent the interview questions in
advance to facilitate the process. After the interview, the instructional designer will make the
suggested changes to the website.
The next type of evaluation will be a one-to-one evaluation. There will be 3 different
assessment tools and processes used for the one-to-one evaluations. The first will be an attitude
and acceptance survey, developed by the instructional designer to assess participants overall

Evaluation Proposal

attitude regarding the DLS language learning program and its success in achieving its goal. This
survey will be created and distributed to all participants via the online survey tool
SurveyMonkey. It will consist of 10 questions in multiple-choice format and it will be sent to
participants email addresses. All participants in the DLS program check their emails daily, so the
administration of the survey will not be an issue. A reminder to complete the survey will be sent
on July 9, letting participants know that the survey period will end on July 15.
The next assessment will be one-to-one interviews with 5 participants in the DLS
program, carried out via an online synchronous meeting using Skype or Google Hangouts due to
the fact that the participants all live in Sao Paulo, Brazil. The participants will be selected from
different departments at Global Enterprises Sao Paulo, to ensure that we get a varied sampling of
the learners. One interview will take place per day, to allow for the time-zone difference as well
as the most convenient time for the participant to take part in the interview. To encourage overall
participation, two $100.00 Amazon gift cards will be raffled at the end of the program
evaluation.
Finally, there will be an interview with the DLS program instructor. Due to the
instructors busy schedule, she will be given a choice to be interviewed either via email, over the
telephone, or via a synchronous online meeting using Skype or Google Hangouts to facilitate the
collection of data.
During the month of July, and especially during the period of July 15-31, the data
collected from surveys and interviews will be analyzed and changes will be made to the DLS
website based on the results. Following is a sample containing 5 questions that were used in each
assessment tool.

Evaluation Proposal
SUBJECT-MATTER EXPERT INTERVIEW QUESTIONS (Sample 5 questions)
(adapted from Tessmer, 1993, p. 58)
1) Is the content accurate?
2) Are the directions clear?
3) Would students be interested in the videos displayed on the website?
4) Are the visual elements appealing?
5) If you could change something in this website, what would it be?
ATTITUDE AND ACCEPTANCE SURVEY QUESTIONS (Sample 5 questions)
1) The DLS program is useful for me.
Strongly Agree Agree Neutral Disagree Strongly Disagree
2) The DLS program is easy to participate in.
Strongly Agree Agree Neutral Disagree Strongly Disagree
3) The DLS program content is appropriate for my level.
Strongly Agree Agree Neutral Disagree Strongly Disagree
4) The software used in the DLS program is easy to use.
Strongly Agree Agree Neutral Disagree Strongly Disagree
5) The DLS program instructor is helpful and approachable
Strongly Agree Agree Neutral Disagree Strongly Disagree
PARTICIPANT INTERVIEW QUESTIONS (Sample 5 questions)
1) What do you think of the DLS program?
2) Do you feel that your instructor has been helpful and approachable? Why/Why not?
3) What do you think of the content displayed in the website?
4) Do you think that your English language ability has changed since you started in the
program? Why/Why not?
5) What is one thing that you would change about the program?

Evaluation Proposal

DLS PROGRAM INSTRUCTOR INTERVIEW QUESTIONS (Sample 5 questions)

1) What do you think of the DLS program?
2) Do you feel that the participants are interested and willing to learn? Why/Why not?
3) Have any students come to you with issues regarding the software used in the DLS
website? If so, what were these issues?
4) Have you seen progress in the participants English language proficiency?
5) What is one thing that you would change about the program?

Summative Evaluation
In the summative stage of the evaluation plan, one-to-one evaluation type of assessment
will be carried out through interviews, surveys, and a post-test. The following timeline will be
observed. Note: United States national holidays will be followed since the instructional designer
resides in the U.S.
TIMELINE SUMMATIVE EVALUATION (August-October 2015)
August 1 20
August 21 25

August 26
August 27
August 28
August 31
September 1

Develop evaluation tools (interviews and

post-test)
Schedule interviews with 10 participants in
the program (August 26 September 9)
Send attitude and acceptance survey to all
participants in the program
Schedule interview with DLS program
instructor (September 10)
Schedule date for English Language
Proficiency post-test (September 11)
Interview Participant 1
Transcribe data
Interview Participant 2
Transcribe data
Interview Participant 3
Transcribe data
Interview Participant 4
Transcribe data
Interview Participant 5
Transcribe data

Evaluation Proposal
September 2
September 3

September 4
September 7
September 8
September 9
September 10
September 11
September 12
September 13 October 31

10
Interview Participant 6
Transcribe data
Send out reminder to take attitude and
acceptance survey to all participants in the
program
Interview Participant 7
Transcribe data
Interview Participant 8
Transcribe data
No interviews
Labor Day Holiday
Interview Participant 9
Transcribe data
Interview Participant 10
Transcribe data
Interview DLS instructor
Transcribe data
Apply English Language Proficiency posttest
Close attitude and acceptance survey
Grade post-test
Analyze data from interviews, surveys, and
post test
Make suggested changes to the DLS
website

Description of Procedure and Tools

For the summative evaluation, attitude and acceptance surveys will be distributed via
SurveyMonkey to all participants in the DLS program. This survey will be the same as the first
one given out to participants in the formative evaluation phase of the program. It will act as a
post-test to see how attitudes have changed after the modifications have been made to the overall
website and materials.
In addition, interviews with 10 participants will be carried out. The same system
previously used for the formative evaluation will take place, since all participants live in Sao
Paulo, Brazil. Therefore, participants will be interviewed via an online synchronous meeting

Evaluation Proposal

11

using Skype or Google Hangouts. Similarly, only one interview will take place per day, to allow
for the time-zone difference as well as the most convenient time for the participant to take part in
the interview. The same initial 5 participants will be encouraged to take part in the process, and
newly selected 5 participants will be added to the participant poll. These 5 new participants will
also be selected from different departments, making sure that there is a varied sampling of the
learners involved in this evaluation. All participants will be reminded that there will be two
$100.00 Amazon gift cards that will be raffled at the end of the program evaluation, to encourage
full participation and engagement throughout the process.
The DLS instructor will also be interviewed, and due to her busy schedule, she will be
given a choice to be interviewed either via email, over the telephone, or via a synchronous online
meeting using Skype or Google Hangouts to facilitate the collection of data.
Finally, on September 11, an English language proficiency post-test will be given to all
participants at the Enterprise Conference Room, with the supervision of proctors who belong to
the staff of Global Enterprises in Sao Paulo, Brazil. The proctors will sign confidentiality
agreements and will be responsible for the security of the material distributed to students on that
day. The test will consist of 20 questions created to assess the overall development of the
participants English language proficiency. These 20 questions will evaluate similar grammar
topics to the ones initially given to participants during their placement test, carried out in the
beginning of the implementation of the program as a pre-requisite for participation. Note: The
placement test occurred before the program evaluation plan was requested.
All participants will follow this procedure in order to take the post-test:
a) Participants will be checked-in to the conference room;
b) Participants will place their belongings in the front of the room and be assigned seats;

Evaluation Proposal

12

c) Tests will be distributed by the proctors, who will also give participants one #2 pencil
and a Scantron sheet;
d) Participants will be given 15 minutes to answer the questions. No participant will be
allowed to leave before the allotted time.
e) At the end of the 15 minutes, tests, Scantron sheets, and pencils will be collected and
participants will be dismissed.
During the months of September and October, and especially during the period of
September 13 October 31, the data collected from surveys, interviews and post-tests will be
analyzed and modifications, in particular new additions to the DLS website content, will be made
based on the results. Following is a sample containing 10 questions from the English language
proficiency post-test.
ENGLISH LANGUAGE PROFICIENCY POST TEST (Sample 10 questions)
1) I ______________ very busy lately
a) have been
b) am
c) been
d) am being
2) The boy ___________ is my best friend.
a) who lives next door
b) that lives next door
c) both a and b are correct
d) none of the above
3) I dont understand _________________.
a) what are you saying
b) what you are saying
c) that you are saying
d) none of the above

Evaluation Proposal
4) I turned off the TV and went to bed early because ______________.
a) I had already seen that movie
b) I already saw that movie
c) I never saw that movie
d) None of the above
5) On no occasion _______________ allowed to enter this room.
a) will you be
b) you will be
c) both a and b are correct
d) none of the above
6) My best friend and I ____________ at Global Enterprises.
a) works
b) work
c) both a and b are correct
d) none of the above
7) My instructor _______________ English for over 10 years.
a) has taught
b) has been teaching
c) both a and b are correct
d) none of the above
8) I dont know _______________ about politics.
a) anything
b) nothing
c) both a and b are correct
d) none of the above
9) ____________________ is this?
a) Whose pen
b) Who pen
c) What the pen
d) Which the pen

13

Evaluation Proposal

14

10) How long _________________ for me?

a) are you waiting
b) have you been waiting
c) both a and b are correct
d) none of the above

Confirmative Evaluation
For the confirmative stage of the evaluation plan, an attitude and acceptance survey, a
new English language proficiency post-test, and a final interview with the DLS program
instructor will be carried out in February of 2016, three months after the summative evaluation
takes place. The confirmative evaluation assessment aims to make sure that the changes made to
the program made it a successful tool that can facilitate communication between the Japanese
and the Brazilian staff at Global Enterprises. After it is confirmed that the program is successful
in achieving its goals, the instructional designer will present the results to the stakeholders at
Global Enterprises, who can then sanction the overall expansion of the program to reach
Brazilian staff members of all English language proficiency levels.
The following timeline will be observed. Note: United States national holidays will be
followed since the instructional designer resides in the U.S.
TIMELINE CONFIRMATIVE EVALUATION (February 2016)
February 1 - 5
February 6
February 8 - 10

Develop evaluation tools (interview and

post-test)
Send attitude and acceptance survey to all
participants in the program
Schedule interview with DLS program
instructor (February 16)
Schedule date for English Language
Proficiency post-test (February 17)

Evaluation Proposal
February 12

February 16
February 17
February 18 28

February 29

15
Send out reminder to take attitude and
acceptance survey to all participants in the
program
Interview DLS instructor
Transcribe data
Apply post-test

Grade post-test
Analyze data from surveys and interview
Make possible changes to DLS website
Meet with program participants and
stakeholders to discuss program success.
Raffle Amazon gift cards

Description of Procedure and Tools

For the confirmative evaluation, attitude and acceptance surveys will be distributed via
SurveyMonkey to all participants in the DLS program. This survey will be the same throughout
the entire evaluation plan, to serve as a tool to measure the changes in attitude that participants
might have throughout the entire program.
In addition, the DLS instructor will be interviewed for the last time using a choice of
email, phone, Skype or Google Hangouts. Her input throughout the entire process is also
important, since she is the one who evaluates the speaking and written portions of the content on
the DLS website. She also can have a good idea of the participants overall progress throughout
the program, especially after changes have been implemented.
Finally, another English Language Proficiency post-test will be given to all participants.
This time, it is in a form of an essay question that will ask participants to write an email to their
counterparts in Japan describing a business issue that they have had recently. The post-test will
be applied in the same manner it was conducted in September, 2015. Participants will take at the
Enterprise Conference Room, with the supervision of proctors who belong to the staff of Global

Evaluation Proposal

16

Enterprises in Sao Paulo, Brazil. The proctors will sign confidentiality agreements and will be
responsible for the security of the material distributed to students on that day.
All participants will follow this procedure in order to take the post-test:
a) Participants will be checked-in to the conference room;
b) Participants will place their belongings in the front of the room and be assigned seats;
c) Tests will be distributed by the proctors, who will also give participants one #2 pencil
d) Participants will be given 50 minutes to write their essay. No participant will be
allowed to leave before the allotted time.
f) At the end of the 50 minutes, tests, and pencils will be collected and participants will
be dismissed.
During the month of February, and especially during the period of February 18-28, the
data collected from surveys, interviews and post-tests will be analyzed and a final assessment
will be made to confirm whether the program was successful in achieving its goals.
At the end, the instructional designer will present the results to the stakeholders at Global
Enterprises, who can then sanction the overall expansion of the program to reach Brazilian staff
members of all English language proficiency levels. On February 29, 2016, the participants in
the program and the stakeholders will take part in a meeting where 2 $100.00 Amazon gift cards
will be raffled among staff members who participated in the entire program evaluation.
Following is a sample essay question from the English language proficiency post-test.
ENGLISH LANGUAGE PROFICIENCY POST TEST (Sample Essay Question)
Write an email to a staff member at Global Enterprises Tokyo describing an issue you have
encountered recently and what you would like this person to do to solve the problem. Use the
newly acquired grammar and vocabulary you have been practicing through the Distance Learning
Solutions Language program as much as possible. You have 50 minutes to complete this task.

Evaluation Proposal

17
Reference

Tessmer, M., (1993). Planning and Conducting Formative Evaluations. London: Kogan Page
Limited.