Final Report On Main Survey

for
OECD’s Programme for International Student Assessment (PISA) in India –
2009+

Submitted By

Ratna Dhamija

Australian Council for Educational Research (India)

February 2011

This report has been commissioned by the World Bank, India. The author is responsible for any
errors and the views expressed in the report.
 CONTENTS
Introduction/Overview: 1

About PISA & PISA in India 1

ACER 1

Objectives of the Project 2

Project Activities 2

A) Training of Personnel
a. School coordinators
b. Test Administrators
c. Coders and Markers
B) Quality Monitoring of Field Activities
a. Himachal Pradesh
b. Tamil Nadu
C) Supervision of Coding Activities

Challenges & Issues 7

ABBREVIATIONS

ACER: Australian Council for Educational Research

CBSE: Central Board of Secondary Education

MHRD: Ministry of Human Resource Development

MoU: Memorandum of Understanding

NC: National Centre

NCERT: National Council of Educational Research & Training

NUEPA: National University of educational Planning & Administration

OECD: Organization for Economical Cooperation and Development

PISA: Programme for international Student Assessment

UAE: United Arab Emirates

WB: World Bank

Final PISA Main Survey
_______________________________________________________________________________
About PISA and PISA - India

The Programme for International Student Assessment (PISA) is a major initiative developed, since the
late 1990, by the Organization for Economic Co-operation and Development (OECD), to provide
international comparative data on the educational performance among the participating countries.
PISA is a system of international standardised study that assesses and compares how well countries
are preparing their 15-year-old students to meet real-life opportunities and challenges in an interval of
three years starting from 2000. PISA assesses three key areas of knowledge and skills – reading
literacy, mathematical literacy and scientific literacy – and has a major focus on one of these literacy
areas each time PISA is administered. In 2000, reading was the main focus, in 2003 it was
Mathematics and Science was the main focus in 2006. And in 2009, reading was again the main
focus.

In 2009 PISA cycle, 65 countries (32 member and 33 non member) participated. In India after
continuous deliberation among the leading educational policy making institutes like MHRD, NCERT,
NUEPA, CBSE and development partners like World Bank, a Memorandum of Understanding (MoU)
was signed between OECD and MHRD regarding India’s participation in PISA in 2009 plus cycle. As it
was late for any country to participate in PISA, a new timeline was developed to accommodate India
and eight other countries (Georgia, Costa Rica, Mauritius, Malaysia, Moldova, Miranda, Malta,
UAE(except Dubai). Initially it was suggested, India was to participate in PISA with four states but on
further deliberations it was decided that only two Indian states (Himachal Pradesh and Tamil Nadu)
will participate in this prestigious activity.

ACER

In PISA 2009 a group of organisations of international repute (ACER Australia, NIER, cApStAn,
Westat) formed a consortium to lead PISA. ACER Australia was the consortium leader for PISA 2009
cycle.

The Australian Council for Educational Research is an independent educational research organisation
established in 1930. With four offices in Australia (a main office in Melbourne, and offices in Sydney,
Perth and Brisbane) and one each in Dubai and Delhi, ACER provides services to the education
community in Australia and overseas. ACER's mission is to create and disseminate research-based
knowledge and tools that can be used to improve learning. In pursuing this mission, ACER undertakes
a wide range of research and development activities and provides services and materials in support of
educational policy-making and improved professional practice.

World Bank contracted Australian Council for Educational Research (India) to support NCERT in
conducting PISA 2009 plus in India in the states of Himachal Pradesh and Tamil Nadu for training of
personnel involve in field administration (school coordinator & test administrator), quality monitoring of
field administration, training of coders and supervision of coding activities.

1
Final PISA Main Survey
_________________________________________________________________________________

Objectives of PISA:
The objective of the study is to provide countries with international comparative data as well as with
national trend data on reading, mathematical and scientific literacy of 15 year old students.
Mathematical Literacy: An individual’s capacity to identify and understand the role that mathematics
plays in the world to make well-founded judgments and to use and engage with mathematics in ways
that meet the needs of that individual’s life as a constructive, concerned and reflective citizen
Reading literacy: An individual’s capacity to understand, use and reflect on written texts, in order to
achieve one’s goals, to develop one’s knowledge and potential and to participate in society
Scientific literacy: The capacity to use scientific knowledge, to identify questions and to draw
evidence based conclusions in order to understand and help make decisions about the natural world
and the changes made to it through human activity.

Project Activities:

A) Training of Personnel

Training of personnel is an important aspect of any large scale like PISA to achieve the standard. In
PISA field operation, School coordinator and Test Administrator play a crucial role to ensure the
quality of data while collecting individual details of PISA eligible students, about the school,
participation of the students, student contextual data and the administration of PISA assessment tools
in the selected schools. During Main Survey (MS), the study was planned to cover around 7000
students from 250 schools (170 in Tamil Nadu and 80 in Himachal Pradesh). As per the PISA norms
the School coordinator (SC) and Test Administrator (TA) should be two different people for better
delivery. In this scenario, every school had a School Coordinator. A Test administrator was to cover
more than one school depending upon the location, schedule and availability. The training
programmes were organised and conducted by ACER in two phases with coordination with National
and State Centre of PISA, first training programme was planned for School Coordinator (SC) followed
by another programme for Test Administrators. The training was given to them at the State Centres.

a. Training of School Coordinator: Total 170 SCs from Tamil Nadu and 80 SCs from
Himachal Pradesh were selected. The process to select was adopted as defined in
the National Project Manager’s Manual. The School Principal from the sampled
school nominated a senior teacher to bear the responsibilities of a School
Coordinator. The Test Administrators were chosen by the State Centres from the
district clusters. It was ensured that they were not related to any students or school
directly or indirectly. For

Tamil Nadu two trainings session was organized, one in Chennai and another in
Coimbatore to accommodate all School Coordinators. Participatory method of training

2
Final PISA Main Survey
_________________________________________________________________________________

was adopted to make the SCs understand about their role and responsibilities in the
project (as per the School Coordinator’s Manual). A translated version (Tamil) and
original version (English) of School Coordinator manual with a check list of activities
was distributed to each of the School Coordinator for better understanding and
facilitation. Similarly, two training sessions were also organized in Himachal i.e., one
in Solan and other in Mandi to cover all the schools.

b. Training of Test Administrator (TA): Total 145 Test Administrators (110 from Tamil
Nadu and 35 from Himachal Pradesh) were selected by State Project Managers. The
TAs’ were from District Institute of Educational Training or Block Resource Centres.
Two training sessions for Tamil Nadu and one training session for Himachal Pradesh
were organized for Test Administrators covering their role and responsibilities
including a practice session. A translated version (regional language) and original
version (English) of Test Administrator manual with a check list of activities also
distributed to each of the Test Administrator for better facilitation.

c. Training of Coder: Though a rigorous process was followed to select the coders,
training of the coders is as important as other activities for the success of PISA. In
India, PISA is administered in three languages i.e., English, Hindi and Tamil. From
each language 20 coders were appointed (12 from reading, 4 from mathematics & 4
from mathematics). All the coders were divided in to two groups i.e., table leaders and
coders. All the coders were first oriented about the PISA and PISA assessment
instruments. Then the subject specific coding procedure was discussed with the table
leaders as they play a very important role in monitoring the quality of coding. Then
three training session was conducted (Reading, Mathematics & Science) to discuss
the coding procedure and issues. Besides the training a practice session was also
organised for all the coders before the main coding exercise to minimize the coding
error. Coding was organised under thorough supervision on daily basis to resolve the
coding related issues. It was the responsibility of ACER India to monitor the coding
procedures and processes and to ensure that all PISA norms were followed by the
coders

B) Quality Monitoring of Field Activities:

The Quality Monitoring for test administration was another responsibility of Australian Council
for Educational Research (India). The guidelines that were adopted to conduct quality
monitoring were as per the specified norms in the PISA Quality Monitor Manual approved by
OECD. The purpose of the monitoring is to document the extent to which testing procedures
in schools meet the defined standards. The selection of schools was made on the basis of the
sampling frame and takes into consideration of:
3
Final PISA Main Survey
_________________________________________________________________________________

 location
 schedule
 type of school
 ease of access

The change of schedule for test administration from one month to 15 days (earlier 1st to 31st
October but changed to 17th to 31st October) lead to a smaller selection of schools than earlier
intended. Six schools from both the states Tamil Nadu & Himachal Pradesh, were finally
selected for Quality Monitoring. They were Co-education schools, four from rural area and two
from urban area. The details of these schools are as below:

Himachal Pradesh

1. Delhi Public School, Jhakari, Dist. Shimla

School Coordinator: Ms Kalpana Anand Test Administrator: Mr Virendra Chauhan

2. Government Secondary School, Birla Dist. Sirmaur

School Coordinator: Ms Neelam Rana Test Administrator: Dr Balveer Sharma

3. Government Secondary School, Mahunag Dist. Mandi

School Coordinator: Mr Jai Kumar Test Administrator: Mr Ghanshyam Khausal

Tamil Nadu

1. Corporation High School, Coimbatore

School Coordinator: Mr H.Sridharan Test Administrator: Ms Kalaiselvi

2. R.S. Polytechnic College, Dist. Dindigul

School Coordinator: Mr Maria John Test Administrator: Ms J. Chitra

3. Government Higher Secondary School, Kuniyamuthur, Dist. Coimbatore

School Coordinator: Ms R.Punitha Selvi Test Administrator: Mr C Jaynathan

Quality Monitoring Indicators: The quality monitor was conducted on the basis of the following
criteria as prescribed in Pisa Quality Monitor (PQM) manual:

 Assessment Material: The test materials reached to the Test Administrators well before
the day of testing except some of the locations in Himachal Pradesh. All the Test
Administrator had organised assessment material very well and was ready for distribution
prior to the
4
Final PISA Main Survey
_________________________________________________________________________________

arrival of the students in test rooms. The test material was distributed as per the defined
guidelines after the students had settled in their seats.

 Test Conditions (place, time & condition): Test was started at the time as was
discussed and decided by the school and test administrator in all schools except Govt
Secondary School, Kuniyamuthur of Dist Coimbatore as the Test Administrator was late.
The material had been re-packed in bags as the school was quite far from the TAs
station. The test rooms were found well lit, spacious and it was possible for the Test
Administrator to move around the room without disrupting students in most of the schools
except Government Secondary School, Birla and Government Secondary school,
Kuniyamuthur of Dist Coimbatore In Government Secondary School, Birla the seating
arrangement was on floor mats which would have probably been okay in case the
students had been informed earlier to bring hard boards that they use at the time of
examinations. Since they were not carrying the same it was difficult for them to write. The
school coordinator had been changed at the last moment and had no instructions and
obviously due to last moment nomination had not attended the training session. The Test
Administrator had not checked with the school coordinator regarding students being
briefed or all other arrangements being managed.

 Unpredictable Events: In the Govt Secondary school, Kuniyamuthur of Dist Coimbatore,

the arrangement for test was not appropriate as it was noisy due to student’s playing
outside and without adequate lights. On intervention from the quality monitor efforts were
made to stop disturbance but the Test Administrator & School Coordinator could do very
little to stop all these disturbances during the test period.
 Use of Test Administrator Manual: All the Test Administrators were aware about their
responsibilities and read the script from the Manual for instructing the students and
informing them about the PISA. The Test Administrators of Govt. Secondary School,
Kuniyamuthur of Coimbatore District did not refer to the manual and neither provided any
guidance to the students. In fact he allowed the school coordinator along with another
teacher to conduct the test.
 Attendance of Students: The average attendance was around 85.28% in all the
schools. Due to the heavy rain fall & bad weather, attendance was very poor in Govt.
Secondary School, Mahunag in Mandi District, It was only 55% as this school covers a
long range of isolated villages in its area of Himachal Pradesh that’s why the students
were unable to reach the school and the test was postponed to 28th Oct 2010.
 Confirming the Students’ Name & Students’ identification Number: It was found that
during the test, in Govt. Higher Secondary School, Kuniyamuthur of Coimbatore, the
students’ information over the cover page of test booklets (which were not correct) were
not checked by the Test Administrator & School Coordinator, during the random check by

_______________________________________________________________________________
5
Final PISA Main Survey
_________________________________________________________________________________

Quality Monitor, it was noticed that the students had not mentioned their information over
the test booklets and immediately brought this point in notice of Test Administrator as well
as School Coordinator. In rest of the schools Test Administrators had checked all the
information over the test booklets of all the students.
 Record if students were present or absent: Yes, It was recorded in all the schools but
in Govt. Higher Secondary School, Kuniyamuthur of Coimbatore, School Coordinator
substituted all the absentees by other students without consulting the State Project
Manager/National centre. On the intervention of the Quality Monitor, Assistant State
Project Manager was contacted and the instructions were received to exclude all
substituted students. The students were asked to leave even though by that time they
had been provided with the test booklets and had already started the writing of test on
test booklets.

 Test Session: The test was started on decided time in most of the schools & Test
Administrators read the first hour of session part of the script exactly as it was written in
their manual also in the beginning of test, all the Test Administrator made sure that the
instructions provided to them during the training and in the manual were followed. Again it
was observed that the Govt. Higher Secondary School, Kuniyamuthur of Coimbatore did
not refer to the manual and neither did he provide any instructions to the students as the
test was conducted. In all the schools, The Test Administrator had asked the students if
they have any question to ask after reading the direction, the example and the answer to
the students for the test session The Test Administrator read the concluding the test
session part of the script exactly as it was written in the manual and in most of the
schools all the students also remained seated until the Test Administrator said they could
leave.

 Student Questionnaire session: In all the schools, all the students completed their
student questionnaire immediately after the Test session except Govt Secondary School,
Kuniyamuthur of Dist Coimbatore as they attempted the student questionnaire first and
then completed the test booklet. In all the school except Govt Secondary School,
Kuniyamuthur of Dist Coimbatore, The Test Administrators confirmed that the student’s
name and their student identification number on the student tracking form matched with
their questionnaires and Test Administrators also read the ‘ Introducing the student
questionnaire’ part of the script exactly as it was written in their manual. The Test
Administrators followed the procedure and read the ‘Ending the student questionnaire
session’ part of the script.

 Observation about Students: Generally in all the schools, students were orderly & co-
operative except Govt Secondary School, Kuniyamuthur of Dist Coimbatore. Few

6
Final PISA Main Survey
_________________________________________________________________________________

instances were observed where the students were looking into each others’ work.
Students were not mentally prepared for the test.

Interventions by Quality Monitor (ACER, India Representative): In Delhi Public School, Jhakari,
Shimla, the Test Administrator went outside of test room for around 15 to 20 minutes. In GSS, Birla the
Test Administrator was treating this test as a routine exam and had not tried to control them so Quality
monitor spoke with the Test Administrator and controlled the situation but it was only for few minutes
as Quality Monitor took steps in its beginning stage. In GSSS, Kuniyamuthur the conduction of test
was somehow deviate the norm by substitute the students in lieu of absent students.

Interview with School Coordinator: All the School Coordinator received very good support from
school management, parents & students. School Coordinator from Corporation High School,
Coimbatore & R.S. Polytechnic College, Dindigul suggested that August, September or first week of
October is also good for such kind tests as the students had exams in month of November. School
Coordinator did not face any problem in preparing the accurate list of students those who were in PISA
eligible criteria.

Challenges and Issues:

Planning and completing tasks as per the time schedule is the most important in an international
activity such as PISA. The standards and procedures need to well understood and followed by the
book by all supporting the implementation of PISA. Generally the management of test material at Test
Administrator level was better. Though it is advised that the Test Administrator may further need
instructions from the state authorities to conduct the administration with a more serious approach and
they have to follow the instructions given to them during the test administration training. The
contribution of most of the Test Administrators was very good but it needs to be improved. They
should be more familiar with the procedures and must understand the importance of International
Assessment Programmes and not have a casual approach.

The contribution of School Coordinators was also good but during the monitoring it was noticed that
some of the coordinators had not attended the training session as they were changed at a later stage.
It needs to be communicated to the school principal that a change of school coordinator is not
advisable after the training session. It was also found that some of the School Coordinator were not
informed before time and received the material related to PISA training after the training programme. I
think we should keep this in mind that during such kind activities all the School Coordinator should be
well informed before the administration of programme. As they are the pillars and ultimate
responsibility goes to them for implementation of programme. It is suggested that a meeting and
training session for the Principals should also be organised so that the importance of the international
participation can be explained to them and following of the standards may be understood by them. It
will help them in better

______________________________________________________________________________
7
Final PISA Main Survey
________________________________________________________________________________
supervision. We also need to be more careful during the translation of test booklets. It should be
correct and simple to understand for school level children as in some of the school students were
unable to understand the meaning of few words.
It is also important to emphasize to all stakeholders the importance of following the PISA norms as
stated in the manuals. This will ensure better management of all processes. Selection of Test
Administrators may be reviewed to ensure that the person appointed is proficient in both the state
language and English.