Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Description of Process
Level 1 Reaction surveys and Level 2 Learning assessments will be given
immediately upon completing the seminar and online course. Level 3
evaluations will take place one month and three months after the initial
course. This will give participants time to implement the things they learned
in the courses. Participants will do a self-evaluation and a supervisor will also
do an evaluation of participants LiveLesson to see if they are utilizing the
tools they learned in the training. In this context, it is difficult to measure the
results of the training. However, results will be measured by looking at
trends in LiveLesson attendance. If attendance trends upward, it will be
inferred that participants are using LiveLesson effectively. If attendance is
trending downward, the training should be evaluated to determine if
teachers need to learn other things to make the LiveLessons more effective
and engagement. See Appendix B for a Gantt Chart of the timing of each
task.
Instruments & Instrument Descriptions
Evaluation will be done at all 4 levels of Kirkpatricks Four Levels
(Kirkpatrick & Kirkpatrick, 2006, p. 21). The purpose of the evaluation will be
to gain information on how to improve future training programs
(Kirkpatrick & Kirkpatrick, 2006, p. 21). According to one new hire, it would
have been better to have the online training before the seminar. The timing
of both will be measured in the reaction sheet to see if this is more than one
persons opinion.
Immediately following the seminar, a smile sheet will be given to
measure reaction, Level 1 of Kirkpatricks model (Kirkpatrick & Kirkpatrick,
2006, p. 21). Participants will be asked to fill it out before leaving the room.
This should ensure a 100% response rate as recommended by Kirkpatrick
(Kirkpatrick & Kirkpatrick, 2006, p. 35). The reaction sheet will be anonymous
and will measure how the participants felt about the training. The reaction
sheet will have questions that use a 3 level evaluation (too soft, just right,
too loud) and 5 point Likert scale as well as a section for comments.
Although it is recommended that reaction sheets evaluate content
(Kirkpatrick & Kirkpatrick, 2006, p.120), questions about the content will be
omitted since this session is an overview. It is not expected that all the
participants questions about LiveLesson will be answered at this time.
See Appendix C for the survey.
Task Who
Seminar Participants
Start Date
Numb
er of
days
End
Date
17-Aug
17-Aug
17-Aug
18-Aug
18-Aug
11
Evaluation
Team
Participants
1-Sep
17Aug
17Aug
17Aug
18Aug
28Aug
7-Sep
1-Oct
7-Oct
Lead Teachers
8-Oct
Analysis of online
Learning*
Self- Evaluation of
Practices*
Evaluation of Practices*
1-Dec
14Oct
7 7-Dec
Practices*
Evaluation of Practices* Lead Teachers
Data analysis of behavior Evaluation
evaluations Team
Analysis of Attendance* Evaluation
Team
Final Report Delivered Evaluation
Team
*Data will be collected during these tasks.
8-Dec
16-Dec
16-Dec
23-Dec
14Dec
10Jan
10Jan
15Jan
Analysis Process
Level I
Reaction Sheet for Seminar. Table 2 gives guidelines for the
interpretation of the data, based upon the average response to each
question, which in turn will be averaged for the category (Kirkpatrick &
Kirkpatrick, 2006, p. 159).
Table 2
Range
1-2
3 3.6
3.7
4.2
4.2 5
Interpretation
The participants did not feel that this part was done well.
There were some good things about this area, but it could be
improved.
Participants had a positive reaction to this area of the training.
The participants felt there was little to change in this area of the
training.
Comments: I felt the instructor assumed that I knew something about this
LiveLesson and I didnt. He didnt make it clear at the beginning what we
were learning about. I had to figure it out as the session continued.
As a result of the low scores for the advanced skills, the evaluation team
decided to interview a sample of teachers (5 of the 20) to see why they were
doing the basic skills and not the advanced skills. Most reported that they
forgot that those things were possible or they did not think they were
necessary. When they learned that they should be doing something, like
logging attendance, they asked another teacher about it.
Recommendations. The surveys and interviews indicate there may be
some information overload at the beginning of the school year. In order to
make sure teachers are fully utilizing LiveLesson, it is recommended that the
advanced portion be taken later into the school year, when teachers have
become comfortable with the basics and are better equipped to make things
more complicated.
Level 4
Data collection. Attendance information is recorded automatically by the
software and can be downloaded into a spreadsheet. If the attendance to
LiveLesson sessions are decreasing on average as time goes by, the
evaluators will need to determine how to adjust the training so that teachers
use LiveLesson more effectively. The school calendar and policy will also
be considered when analyzing these trends. For example, if a mandatory
LiveLesson policy is implemented for certain students, those students will
not be included in the final count.
Results. For most teachers, attendance was high at the beginning and
began to dip. However, after a few weeks it began to increase. Chart 1
shows a sample of two classes.
Chart 1
1 2 3
4 5 6 7
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Executive Summary
To: Members of the School Board
From: Evaluation Team
As a part of your ongoing process to improve the onboarding process, you
asked us to do an evaluation of the LiveLesson training for newly hired
teachers to determine ways that this training could be improved. Since
LiveLesson is one of the main ways that teachers communicate instruction
to students, it is important that they are well trained in this medium.
Currently, new teachers attend a one hour seminar that introduces
LiveLesson and then take an online course that consists of three modules:
Basic LiveLesson, Policies for LiveLesson and Advanced LiveLesson.
The evaluation was based on Kirkpatricks four levels of evaluation: reaction
to the training, learning from the training, transfer of learning to actual skills
and final results (Kirkpatrick & Kirkpatrick, 2006, p. 21).
After the seminar and online training, participants were surveyed about their
reactions and given a traditional test to assess their learning. Sample
surveys and tests are included in this document. One month and three
months later, participants evaluated their own use of LiveLesson and lead
teachers evaluated the participants use of LiveLesson. Finally, the
students attendance in LiveLesson sessions was analyzed.
The way the seminar was done, some participants were left confused and
there was some problems with the technology and the participants ability to
see the screen clearly. One recommendation of this committee is that the
teachers take this seminar with computers available to them. Perhaps they
could even take the seminar via LiveLesson so that they can really
understand the purpose of LiveLesson and how it will look to their
students. This would also eliminate the visual issues since each teacher
would be in front of his or her own screen.
Based on the tests and the self-reports of skills learned, it appears that there
is too much information in the initial module. Participants learned the basics
of LiveLesson, but they were not implementing many of the advanced
features even three months later. We recommend that the advanced
training be postponed until the teachers become proficient with the basic
features.
Student attendance starts high and then begins to decrease. However,
especially between months 1 and 3, the attendance levels off and then
begins a slow increase. This indicates to us that the training is effective and
students are finding it beneficial to attend the LiveLessons.
In summary, our recommendations include
1) Teaching the seminar in LiveLesson.
2) Postponing the advanced session until later in the school year.
We think that these two simple changes could cause your teachers to use
LiveLesson even more effectively so that more students benefit from the
LiveLesson experience.
We have enjoyed working with your teachers. We look forward to working
with you and them in the future.
Sincerely,
Amy Keyser
References
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs:
The Four Levels (Third ed.). San Francisco: Berrett-Koehler Publishers, Inc.
What is Connections Academy? (2015). Retrieved from
http://www.connectionsacademy.com/online-school on September 18,
2015.
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Introduction to LiveLesson
Sessions
LiveLesson Server Clusters
Accessing Your LiveLesson
Room
Pod Demonstration
Uploaded Content
Meeting Room Management,
Microphone, and Layouts
LiveLesson Room Maintenance
Module 2: Connections
Educations Policies for
LiveLesson
too cold.
right.
What could have been done to make this session more comfortable?
The instructor spoke
too soft.
right.
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
What more could the instructor have done to help you learn how to use
LiveLesson?
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
Yes
No
1 2 3 4 5
Not really
Somewhat
Very much
Not really
Somewhat
Very much
Not really
Somewhat
Very much
Not really
Somewhat
Very much
Not really
Somewhat
Very much
Too short.
Just right.
Too long.
Too much: I feel overwhelmed.
Just right: I think I can lead a
LiveLesson.
Too little: I still have too many
questions.
Boring. I could barely stay awake.
Okay.
Interesting. I was fully engaged!
Yes
No
Disagree
Not sure
Agree
Disagree
Not sure
Agree
Free response
Free response
Free response
1)
2)
3)
4)
Neve
r
What would you like to learn more about in terms of using LiveLesson?
Would you like to receive more training in LiveLesson? If so, how would
you like to receive it. Number your responses with 1 being the best way:
___ A LiveLesson session.
___ A LiveLesson recording.
___ An online training like the original training.
___ A seminar.
___ one on one training with another teacher.
Or
__ I would not like to do more training on LiveLesson.
Online
trainin
g
Asked a
coworker
Experimenti
ng