Sei sulla pagina 1di 73

LRMDS

Assessment and Evaluation


(A&E)

Evaluating Learning
Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS

lrmds pub 1.pub

ro nga
ib
l
a
g
n
n
a
m
Ania
g nga
in
b
u
i
it
y
a
g
makagaya agbalin daytoy a
o
agbasa, tapn a, ken makatulong
n
paset ti biag panagdur-as na,
daytoy ti
bro
li
a
y
o
t
y
a
d
t
ket nasayaa

LRMDS Assessment and Evaluation: Evaluation of Resources

This session is intended to provide


the basic information and
instructions relating to criteriabased evaluation of resources.
Participants will develop
understanding of the Assessment
and Evaluation guidelines,
standards and specifications as
they relate to the evaluation of
learning, teaching and professional
development resources.
LRMDS Assessment and Evaluation: Evaluation of Resources

Enabling Objectives
Explain why having specifications and standards are
essential to doing an evaluation of learning/teaching
resources
Give a brief description of the contents of the following
areas for which specifications have been developed:

IPR management
Educational Soundness
Educational Quality
Social content
Technical

LRMDS Assessment and Evaluation: Evaluation of Resources

Terminal Performance
Objective
To evaluate at least one resource and
prepare a Request Brief for the action to be
taken on the said resource.
Apply the appropriate evaluative criteria and
instruments in the evaluation of specific
resources.

LRMDS Assessment and Evaluation: Evaluation of Resources

Enabling Objectives
Explain the purpose of the Request Brief and
identify its key contents
Define the key terms/concepts in the LRMDS
A & E system Guidelines

LRMDS Assessment and Evaluation: Evaluation of Resources

Content
The importance of standards and
specifications for evaluation
The LRMDS Assessment & Evaluation
System at the regions, divisions and lead
school levels
Processes, workflow, personnel
Evaluation specs, guidelines and
instruments
LRMDS Assessment and Evaluation: Evaluation of Resources

Content
Preparation of the Request
Brief
Preparation of the Evaluation
Plan
Evaluation of resources using
appropriate criteria and
instruments
LRMDS Assessment and Evaluation: Evaluation of Resources

Introduction to the concepts of Evaluation,


Standards, Specifications

LRMDS Assessment and Evaluation: Evaluation of Resources

Introduction to the concepts of Evaluation,


Standards, Specifications
Activity 1
Aka Mirror, mirror on the wall, which
goody is the tastiest of all?
(Refer to activity sheet #1)

LRMDS Assessment and Evaluation: Evaluation of Resources

10

Activity 1
Instructions:
1.Divide the participants into three groups.
2.Give each team a bag of goodies, e.g.,
(banana cake, hopia and other snack
food, at least 3-4 kinds, plastic spoons
and forks, small paper plates)

LRMDS Assessment and Evaluation: Evaluation of Resources

11

Activity 1
The team should evaluate the food items and agree on a
recommendation for each item.
Which of the items should be recommended for inclusion in
the school cafeteria menu?
Are there snack items which may need reformulation
before being considered for cafeteria distribution?
Each team must have a consensus on their
recommendations. The recommendations of all the
groups will be consolidated.
Each team should have a convener and a documentation
officer.
LRMDS Assessment and Evaluation: Evaluation of Resources

12

Activity 1
Document your process and write your recommendations.
The assigned presenter will present the report of the group
to the plenary.
The report must contain the following information:
(1)A description of the processes that the team used in
evaluating the food items and in agreeing on its
recommendations;
(2)The problems, issues which the group encountered in
the process of evaluation;
(3)The group recommendations based on the evaluation.
LRMDS Assessment and Evaluation: Evaluation of Resources

13

Processing
Discuss the processes that the groups employed.
How did each group conduct the evaluation?
Intrinsic to evaluation are standards and the use of criteria
to measure standards.
What were the criteria and standards used by the
groups?
Did the group try to evaluate without agreeing on the
standards and criteria first?
Why was it important to agree on the standards and
criteria first?
LRMDS Assessment and Evaluation: Evaluation of Resources

14

Processing
Would your work have been easier if the
processes and persons responsible for the
processes were defined, if the areas for
which standards or specifications were
defined?
How did you resolve your differences?

LRMDS Assessment and Evaluation: Evaluation of Resources

15

Evaluation, Standards and


Specifications
What is evaluation?
Evaluation is the systematic determination
of the merit, worth, and significance of
something or someone; fit for purpose;
using criteria against a set of standards

LRMDS Assessment and Evaluation: Evaluation of Resources

16

Evaluation, Standards and


Specifications
A standard is an established norm or
requirement.
Standards are developed or formulated by
professional groups, associations, unions,
guilds who have a stake in the product or
service being evaluated.

LRMDS Assessment and Evaluation: Evaluation of Resources

17

Standards
Sometimes the standards are formulated by
private groups through formal consensus.
In many instances, especially when public welfare
is concerned, government through its
instrumentalities formulate the standards for the
evaluation of a product or service and the
standardization process is by edict, that is it is
established by law or an executive issuance.

LRMDS Assessment and Evaluation: Evaluation of Resources

18

Specifications
A specification is an explicit set of
requirements to be satisfied by a material,
product, or service.
A specification is a type of a standard. It
provides the necessary details about the
specific standards required.

LRMDS Assessment and Evaluation: Evaluation of Resources

19

LRMDS Assessment and


Evaluation System
The Assessment and Evaluation System is for
identification of materials to be procured,
redeveloped or reproduced is designed to
support schools, divisions, regions and
central office:
in identifying learning, teaching and
professional development resource needs
reviewing and recommending resources for
acquisition and procurement
LRMDS Assessment and Evaluation: Evaluation of Resources

20

LRMDS Assessment and


Evaluation System
identifying existing resources for
reproduction and or redevelopment
evaluating teacher developed
materials for local use and shared
distribution via LRMDS

LRMDS Assessment and Evaluation: Evaluation of Resources

21

Activity 2
(Refer to Activity Sheet #2)

Activity Instructions:
The activity is to level understanding of basic
terms, concepts, processes and purposes
for evaluation by the group before the
evaluation of actual resources is
undertaken.
Modified Jeopardy game where the
answer is not really the answer and the
question is really the answer.
LRMDS Assessment and Evaluation: Evaluation of Resources

22

Jeopardy
Read the Guidelines and the Handouts to
get the definitions of terms and or
concepts that you will encounter in
evaluation of resources.
Work in pairs. Each pair should write on one
side of the meta strip the definition of the
term; on the other side, write the question
to which the definition is the answer.
LRMDS Assessment and Evaluation: Evaluation of Resources

23

Jeopardy
For example:
Is an opening, or rupture, in a planet's surface
or crust, which allows for the release of hot ash,
and gases.
The answer is A VOLCANO
Curriculum, subject matter, skills and
processes, age and stage of schooling, cultural
inclusivity and language of instruction.
The answer is Educational Specs
LRMDS Assessment and Evaluation: Evaluation of Resources

24

Define/explain
Accessibility Specifications
Intellectual Property Rights Specifications
Storage and Distribution
Publication and Use
Technical Specifications
Assessment and Evaluation
QA Framework
LRMDS Assessment and Evaluation: Evaluation of Resources

25

Jeopardy/ Matching Game


Work as a group.
You are given meta strips with definitions, terms
and explanations.
Read and discuss with your group as to where the
description belong or best describe the terms
posted on the board.

40 minutes
LRMDS Assessment and Evaluation: Evaluation of Resources

26

Terms

Accessibility
Specification

5
6

Description

Geospatial, physical and intellectual ability,


special needs: physical, intellectual differences,
flexible operation and presentation.
Intellectual
Author, contributors, copyright, access and
Property Rights
distribution, equitable: negotiation is balanced
specification
between rights owner and user.
Storage and
Version control for management and
distribution
maintenance, meta data, version control,
discovery, publish in varied formats.
Publication and use File types, formats for reproduction and usage
permissions, communicated information
contains, educational purpose, format.
Technical
Format, file size, authoring software, user plugSpecifications
ins, players, software interoperability.
Assessment and
Identifying learning, teaching and professional
Evaluation
development resource needs, reviewing and
recommending resources for acquisition and
procurement.
QA framework
Supports LRMDS personnel to identify and
consistently implement required actions to
LRMDS Assessment
andagreed
Evaluation: standards
Evaluation of Resources
achieve
and specifications.
27

Quick Processing of the activity


Purpose of the activity: assist the
participant manage his learning through
a collaborative activity.
Establish a common
understanding/working definition of terms
to be used in subsequent activities.

LRMDS Assessment and Evaluation: Evaluation of Resources

28

Introduction to QA and Criteria


based evaluation

LRMDS Assessment and Evaluation: Evaluation of Resources

29

What is Quality Assurance?


Quality
refers to the totality of characteristics of a product
or service that bears on its ability to satisfy stated
and implied needs.
Principles
provide the frame of reference for achievement of
quality.
Measures & requirements
base level descriptors.
Criteria
operational requirements for quality.
Indicators
30
concrete objectives for criteria.

Ensuring Quality
LRMDS Quality Assurance Framework describes the
parameters to ensure that all learning, teaching and
professional development resources that are catalogued
and accessed or located via the LTMDS web portal are
of high educational and technical quality.

LRMDS Assessment and Evaluation: Evaluation of Resources

31

Ensuring Quality
QA is a coordinated process for deliberately bringing about
the attainment of the desired quality. It is a systematic
process of monitoring all management, organizational,
productivity and services of the LRMD systems against
standards of quality in order to raise performance.

LRMDS Assessment and Evaluation: Evaluation of Resources

32

Ensuring Quality
LRMDS Quality Assurance Framework describes the processes that ensure
that learning, teaching and profession development resources are:

Relevant to the needs and expectations of the users;


Accessible by the range of technology available to the users;
Usable and functional, and;
Pedagogically sound and cohesive.

LRMDS Assessment and Evaluation: Evaluation of Resources

33

Purpose
The LRMDS Quality Assurance Framework has been
developed to assist:
LRMDS Managers to manage, maintain and
continuously improve access to quality resources,
LRMDS development and production teams to comply
with agreed standards and specifications for educational
resources.
Key LRMDS personnel within Regions and Divisions to
implement QA processes to ensure the continued
provision of quality services and products
Schools and teachers to be able to identify quality
resources in relation to educational need
LRMDS Assessment and Evaluation: Evaluation of Resources

34

Quality Framework
Area of Quality:

Principles:
Characteristics
Criteria &
indicators

Specifications
and/or
Requirements

Guidelines
Procedures
Instruments
(tools/templates)

Review

Application

A process of continuous improvement


35

Measures & Requirements


LRMDS Quality Assurance measures &
requirements:
Clearly documented and transparent processes
Well defined standards, specifications and
evaluation criteria
Well trained personnel
Capacity of the LRMDS system to respond to
user need.
High quality products and services
LRMDS Assessment and Evaluation: Evaluation of Resources

36

Principles
The main principles underlying the LRMDS quality assurance processes
are:

User centred

There is active and integrated involvement of LRMDS users at all


levels school, division and region.

Responsive

LRMDS product and service delivery is informed by user


requirements and feedback for each of the LRMDS sub-systems.
LRMDS employs the use of the most appropriate media for the
content, taking into account the profiles of schools, teachers, learners
and access to technologies.

Standards based

LRMDS inputs, processes and outputs conform to agreed quality


standards and specifications.
LRMDS Assessment and Evaluation: Evaluation of Resources

37

Under the microscope:


LRMDS QA

LRMDS Assessment and Evaluation: Evaluation of Resources

38

LRMDS Assessment & Evaluation QA

LRMDS Assessment and Evaluation: Evaluation of Resources

39

Activity 3
(Refer to activity sheets #3)

Evaluating Resources
Purpose of the Evaluation: To select resources to be:
A. Catalogued in the LRMDS portal
B. Uploaded to the portal and catalogued
Types of Resources:
Print
Digital print
Online websites
Video based
Interactive resources
LRMDS Assessment and Evaluation: Evaluation of Resources

40

Instruments used and their purposes


Refer to the instruments and point out to
the mandatory requirements stipulated in
the instrument.
Ask participants to read them out aloud.

LRMDS Assessment and Evaluation: Evaluation of Resources

41

Evaluation Streams
Explain to the participants that in LRMDS, the
instruments will be used in various streams of
evaluation, namely,
Evaluation of existing resources in existing
format for reproduction and cataloguing: digital
and non digital
Evaluation of existing resources for
redevelopment: digital and nondigital
Evaluation of teacher-developed resources
Evaluation of On-line resources and sites
LRMDS Assessment and Evaluation: Evaluation of Resources

42

Activity 3
Determine which of the Evaluation instruments
should be used.
Refer to Section 6 Guidelines and Processes for
LRMDS Assessment and Evaluation.
Use the available copies of the instruments for
the actual conduct of the Evaluation
Each group evaluates at least 1 resource using
the instruments.

LRMDS Assessment and Evaluation: Evaluation of Resources

43

Activity 3
All resource evaluations should commence with the IPR
review. When many resources belonging to a series are
to be evaluated then the IPR can be completed once for
all.
Any DepED owned and copyright resources for which
content errors, accuracy issues, editorial issues or layout
issues are identified can be fixed and revised and then
uploaded. If the IPR allows. The requirement to revise
and correct the issues would form part of the Evaluation
recommendation.

LRMDS Assessment and Evaluation: Evaluation of Resources

44

Activity 3
Be prepared to make a specific
recommendation at the end of your
evaluation.
As you apply the instruments to specific
resources, note all criteria and instructions
which were problematic.

LRMDS Assessment and Evaluation: Evaluation of Resources

45

Why do we need specifications?


To determine:
Quality and educational value
Fit for purpose
Access and Use
Management, maintenance, storage
Publication, discovery & distribution

LRMDS Assessment and Evaluation: Evaluation of Resources

46

Specifications
Educational

curriculum, subject matter, skills and


processes, age and stage of schooling,
cultural inclusivity

Accessibility

geospatial, physical and intellectual


ability

Technical

format, file size, authoring software,


user plug-ins, players, software
interoperability

IP & rights
management

author, contributors, copyrights, access


and distribution, use

Storage & distribution version control for management and


maintenance, access
Publication & use

file types, formats for reproduction and


use.
47

Achieving quality
No LR, TR, PDM resource is likely to encompass
all quality principles in its own right as many of
these are realised in the relationships between
teacher, learner, resource and learning
environment.
Usually, the principles and associated criteria will
consist of quantitative and qualitative elements.
Thus a concept of 'reasonableness' on balance is
used in the application of the criteria and applied
throughout the quality assurance process.
48

LRMDS Assessment and Evaluation: Evaluation of Resources

49

LRMDS Assessment and Evaluation: Evaluation of Resources

50

Criteria
Criteria are focused requirements for
quality that are close to the operational
level though they do not necessarily
contain a specific scale.
Testable statements formulated on the
basis of accepted statements
They can express minimum requirements
(statutory conditions) and best practice
(requirements that must be fulfilled)
LRMDS Assessment and Evaluation: Evaluation of Resources

51

Indicators
Indicators are:
Concrete and measurable
Qualitative or quantitative
Express central characteristics (e.g. of
resource to evaluated by the criteria)
Indicators should altogether provide a
picture of whether the resource is in
compliance with the criteria.
LRMDS Assessment and Evaluation: Evaluation of Resources

52

Criteria & Indicators Examples


Non print: Criteria

Indicators

Content stimulates and promotes critical


thinking

4. More than 80% of the content


stimulates and promotes critical
thinking
3. 60-80%
2. 40-59%
1. Less than 40%

Graphics/colors/sounds are used for the


appropriate instructional reasons

4. 90-100% is appropriate
3. 80-89%
2. 70-79%
1. Less than 70%

Content is logically developed and


organized

4. 90-100% is logically developed


and organized
3. 80-89%
2. 70-79%
1. Less than 70%

LRMDS Assessment and Evaluation: Evaluation of Resources

5353

Criteria & Indicators Examples


Non print: Criteria

Indicators

Content stimulates and promotes critical thinking


This might occur in many ways, but the main point is that the activity requires a
cognitive effort, not simply a chance selection.
Graphics/colors/sounds are used for the appropriate instructional reasons.
Does the graphic style, use of color and sound match the content and the target
audience?

Content is logically developed and organized


The underlying question is: Can you understand the apparent structure of the
knowledge domain and how it represented to the user? In random sequence
structures, the logic is determined by the user and it is difficult to say that it is not
appropriate from just a review.
LRMDS Assessment and Evaluation: Evaluation of Resources

54

QA Elements A & E example


Structures

Inputs

Processes

Outcomes &
Products

School Learning Resource


Committee

School Report Card

Application of LRMDS
specifications &
criteria/checklists to:

Evaluated & recommended


resources for cataloguing

LRMDS Division Quality


Assurance Team

Annual Implementation
Plan and School
Implementation Plan

Region LRMDS Quality


Assurance Team

LRMDS online catalogue

Trained evaluators

LRMDS Trainers

Existing resources

identify learning,
teaching and
professional
development
resource needs
evaluate teacher developed
materials for local use and
shared distribution via
LRMDS

Reproduction
Redevelopment
Access to quality resources
via LRMDS online catalogue

Analysis of identified school


needs and completion of LR
Plan, production of DEPD
and REDP

LRMDS Assessment and Evaluation: Evaluation of Resources

55

Ensuring quality- Example


LRMDS Assessment & Evaluation system
Process

Core Quality
objective

Identifying the Identifying


responsible
methods or
actors
instruments
that can be
used
to assure
quality

Measuring the
success of the
quality
objective.

Assessment &
evaluation system
effectively &
efficiently
assesses the
needs of target
users

There are LR
plans at the
region, division
& school levels
that respond to
the needs of
target users

School LR
Teams

LR Assistance
Plans of
Reg/Div reflect
consolidated
needs of
School LR
plans

Division
LRMDS Team
& Trainers

Convergence
table to
compare LR
needs identified
with Reg/Div
LR Assistance
Plans

Division
LRMDS
Manager
LRMDS Assessment and Evaluation: Evaluation of Resources

56

Evaluation planning
What is the purpose of the evaluation?
Need for science resources to support teaching and
learning in grade/year.
Science resources exemplifying constructivist learning
principles.

What is the source of the resources?


Within DepED?
External?

How are they to be reproduced & distributed?


Offline?
Online?
Online and offline?
LRMDS Assessment and Evaluation: Evaluation of Resources

57

Evaluation planning
Answers to these questions should determine how
the criteria and outcomes are weighted.
E.G.
A very sound resource that is highly instructivist and
didactic will not fulfill the need for a resource based on
constructivist principles but it can still be a good
resource.
Or a good resource with typographical errors may be able
to be fixed.
Or a good resource in an older version of flash may be able
to be redeveloped.

LRMDS Assessment and Evaluation: Evaluation of Resources

58

Criteria and indicators


Remember to determine that the purpose of
the evaluation is understood by all
evaluators.
Are there options for the outcomes of the
evaluation? For example issues can be
fixed?

LRMDS Assessment and Evaluation: Evaluation of Resources

59

LRMDS A & E Evaluation


Instruments
Intellectual Property Rights
Educational Soundness
Educational Quality Evaluation for:
- Print
- Charts, posters, drill & flash cards
- Non-print
- General reference
Technical: usability, interoperability, offline digital
Accessibility Evaluation (SPED)
Online site evaluation
LRMDS Assessment and Evaluation: Evaluation of Resources

60

Evaluation Planning: Which instruments? What


criteria, if any, is mandatory?
Evaluation Planning

Possible scenarios

What is the purpose of the Evaluation?

To procure science learning resources for


Grade V to be distributed to all schools in
Region
To procure professional development materials
for elementary science teachers

What resources are to be evaluated?

DepED bureau produced.


Foreign Assisted project developed
Commercial

What is the format of the resources

Print
Non-print (digital offline or online)
Online site

What funds are available for procurement?

School MOOE; FAP e.g. SPHERE


Regional funds

What expertise is required to conduct the


evaluation?

Science, elementary education, evaluation,


technical

Application of Evaluation Instruments


Evaluation Scenario
Purpose: To procure science learning resources
for Grade V
Source: Foreign Assisted project developed &
Commercial
Format: print
Funds: SPHERE & Regional funds
Evaluator expertise: Elementary science and
evaluation experience.
LRMDS Instruments:

Evaluation Areas
Intellectual Property Rights
Educational quality

Intellectual Property Rights checklist:


Verify ownership, Copyright, Conditions of
Use.
Educational Soundness General checklist.
Determine the general qualities in relation to
the purpose and whether to proceed with a
more extensive evaluation.
Educational Quality: Rating sheet for Print
resources. Determine suitability for use in
schools, accuracy and currency of all content
coverage, format, presentation & organization

Application of Evaluation Instruments


Evaluation

Using the instrument


criteria

Intellectual Property Rights checklist:


Verify ownership, Copyright, Conditions
of Use.

Finding: DepED copyright, but contains


some third party material.
What should be recommended?

Educational Soundness General


checklist. Determine the general
qualities in relation to the purpose and
whether to proceed with a more
extensive evaluation.

Finding: Identified some content


inaccuracies in Criteria 1 and 5.

Educational Quality: Rating sheet for


Print resources.
Determine suitability for use in schools,
accuracy and currency of all content
coverage, format, presentation &
organization

Finding: Passed all factors but reported


errors of fact, grammatical,
typographical

What should be recommended?

What should be recommended?

Application of Evaluation Instruments


Evaluation

Using the instrument criteria

Intellectual Property Rights checklist:


DepED copyright, but contains some
Verify ownership, Copyright, Conditions of Use. acknowledged third party material (diagrams,
pictures, excerpts of text).
Recommendation: Reproduce only in existing
format.
Educational Soundness General checklist.
Determine the general qualities in relation to
the purpose and whether to proceed with a
more extensive evaluation.

Identified some content inaccuracies in

Educational Quality: Rating sheet for Print


resources. Determine suitability for use in
schools, accuracy and currency of all content
coverage, format, presentation & organization

Passed all factors but reported errors of fact,


grammatical, typographical
Recommend: All errors are fixed, digitize and
reproduce and distribute.

Criteria 1 and 5.*


Recommendation: Do not distribute or
reproduce in current format. If the content
errors are not in 3rd party material then
conduct Ed Quality review, do not reproduce
unless all content inaccuracies have been
fixed.

* If commercial resource contains errors or inaccuracies then report errors but do not
proceed with further reviews.

Activity 4

What is the Request Brief?


What purpose does it serve?
Who prepares the request brief/
What are the major parts or components
of the request brief?

LRMDS Assessment and Evaluation: Evaluation of Resources

65

Request Brief
A Request Brief is a written explanation given to the development and production
team by a requesting party or client to
describe the specific service required and
the general design requirements of the
resource to be developed or produced.

LRMDS Assessment and Evaluation: Evaluation of Resources

66

Request Brief
The purpose of this document is to describe
in detail the service required and the
design requirements of a client/user for an
existing resource that is to be reproduced,
digitized, or redeveloped and for the
design of new resources.

LRMDS Assessment and Evaluation: Evaluation of Resources

67

Prepare a Request Brief


Instructions:
In pairs/or triads prepare the request brief for the
materials you have evaluated.
All the resources evaluated in all the streams of
evaluation must have a corresponding Request Brief.
After the pairs have completed the Request Brief, they
should exchange Request Briefs with another pair.
The task of the pair or triad at this stage is to REVIEW
the completed Request Brief of their peers for the
following:

LRMDS Assessment and Evaluation: Evaluation of Resources

68

Check the Request Brief


Completeness
Correctness
Clarity
Exchange comments, feedback on each
others Request Briefs.
Make the necessary adjustments in the
briefs.
Compile all the accomplished instruments
and request briefs
LRMDS Assessment and Evaluation: Evaluation of Resources

69

Processing
Ask the participants to probe the following:
Is the purpose for which a Request Brief is
prepared understood?
What areas/parts of the Request Brief did the
participants have difficulty with?
Determine whether the difficulty can be
addressed by presenting the materials for the
second time

LRMDS Assessment and Evaluation: Evaluation of Resources

70

LRMDS Assessment and Evaluation: Evaluation of Resources

71

Whatever the
cost of our
libraries, the price
is cheap
compared to that
of an ignorant
nation.
Walter Cronkite

LRMDS Assessment and Evaluation: Evaluation of Resources

72

LRMDS Assessment and Evaluation: Evaluation of Resources

73