DESIGNING ACHIEVEMENT TESTS FOR SIXTH TO TENTH GRADES IN STATE SCHOOLS

Gilma Zuiga, Viviana Mndez, Tania Paola Tinoco, Valentina Villanueva Universidad Surcolombiana-Neiva Two facts have affected the teaching of English in Colombia recently. They are: the coming out of Lineamientos Curriculares en Idiomas Extranjeros (1999), and the inclusion of English as a compulsory component in the ICFES exam. These facts have motivated teachers in state schools to express their desire to have tests that could be a source of feedback in order to improve teaching and learning in their schools. That is why, we have decided to undertake this task and design achievement tests for sixth to tenth grades that focus on the achievement indicators stated for those grades at state schools. This is an ongoing project, and presently, we have designed, piloted and analyzed the test designed for sixth grade. The next step is to continue to prepare and pilot the tests for the next grades. The theoretical framework, and the steps followed to design and validate the tests are presented in this document.

WHY TO TEST
Achievements tests assess what has been achieved or learned from what was taught in a particular course. The purpose is to determine what the student has learned. Achievement tests can provide assessment for teachers and learners. The assessment tasks is nonthreating, allowing learners opportunities to demonstrate what they know and do not know, and providing useful feedback both for the learners and for their teachers. Therefore, the primary function of the achievement tests is instructional, even though their results could also have an administrative function. In an instructional function, achievement tests can help in diagnosing students learning, give evidence of progress, provide feedback to the respondent, or even give information about the quality of teaching. This information can serve as a check for the relationship between what is taught and what is learned. It could, as well, be a valuable source of information to find out weaknesses and strengths of English programs (Cohen, 1994; Bachman, 1990; Delgado, 1996; Estevez, 1996). The results of achievement tests could be useful for administrators, who may use their results for promotion, placement, or certification. Low results could find the causes in staff qualifications which may lead to teacher development schemes. Other causes may include poor resources, lack of students motivation and poor administrative support. The purpose of designing the achievement test is to improve teaching and learning. This assessment can inform the teacher and influence instruction. Teachers can learn about their students, about themselves as teachers, and about the impact of the instructional program and then joint actions could be taken to tackle problems raised. Students could also take actions after recognizing problem areas and work on their weaknesses.

WHAT TO TEST
Canale and Swain provided a theoretical framework for communicative competence that has influenced the design of assessment instruments over the past years. Tests, then began to focus on one or more of the four components making up the construct of communicative competence, namely sociolinguistic, competence, discourse competence, grammatical competence, and strategic competence. For the purposes of these tests we will be using a version of the above theoretical framework, proposed by Bachman (1990) and Cohen (1994). The term used is communicative language ability, as it provides a basis for the development and use of language tests. It recognizes that the ability to use language communicatively involves both knowledge or competence in the language, and the capacity fro implementing or using this competence. Therefore what will be assessed is communicative ability. Part of communicative ability is sociocultural ability, sociolinguistic ability, grammatical ability, textual ability, and strategic ability. Sociocultural ability: It relates to the appropriateness of the strategies selected for realizing communicative functions in given contexts, like (1)the culture involved, (29 the age and sex of the speakers, (3) their social class and occupations, and (4) their roles and status in the interaction. It also evaluates the amount of information, whether too much or too little and the relevance and clarity of the information supplied. Sociolingusitic ability. It deals with the linguistic forms that respondents use to express the intent of the communicative act. It assesses the respondents control over the language forms used to realize (sorry, very sorry, excuse me). , their control of register (academic or casual), or the formality of the utterance. Grammatical ability: Vocabulary, morphology, syntax, and phonology are assessed. The focus is on clear cases of error in form as well as style. Textual ability: It includes the knowledge of the conventions fro joining utterances together to from a text, which is essentially a unit of language structured according to rules of cohesion and coherence. Strategic competence. That is strategies used to compensate for a lack in some language area. This ability cannot be easily observed, however it will be broken down into three components: respondents set communicative goals, plan their use, and execute the plan.

HOW TO CONSTRUCT THE TESTS

According to Estevez (1996) and Cohen (1994), the place to start in designing an assessment instrument is by making an inventory of the objectives that are to be assessed. This involves distinguishing broad objectives from more specific ones. In our case we will be talking about terminal objectives, since state schools nowadays plan the syllabus of the courses in terms of terminal and enabling objectives (Brown, 1994) (logros e indicadores de logro). The steps are 1. Collect the syllabus of the courses from four state schools and make checklists The teachers checked off terminal and enabling objectives that were common in the four schools Important terminal objectives are distinguished from marginal ones for the task.

Objectives are weighed in order to obtain a representative sampling of a learners language behavior. The most emphasized objectives are identified in the courses as well as those of greatest utility value for the students 2. Collect notebooks and portfolios of students used in classes 3. Collect textbooks and indicate material covered during the courses. 4. Interview teachers in four state schools to find out the most common expectations of teachers in relation to the achievement tests. STATING THE SKILLS BEING ASSESSED According to the Lineamientos Curriculares, a communicative methodology is expected to be taught at schools in Colombia, therefore the tests are designed to assess the four linguistic skills: listening, speaking, reading, and writing. STATING ITEM TYPES Some item types stated for the ICFES exam will be developed. Other type of questions are added as the achievement tests developed in this project intends to assess listening and speaking as well. The following were used as types Item type Skill Sentence construction Understanding pictures Incomplete texts Paragraph Construction Incomplete dialogues Understanding situations Text comprehension Personal questions Fill in charts Write texts Reading Speaking Listening reading reading reading Speaking Reading Speaking Listening Writing

Communicative Competence Grammatical Textual Grammatical Sociolinguistic Sociolinguistic Textual Grammatical Textual Sociolinguistic Sociocultural Scoiolinguistic Sociocultural Textual Sociocultural sociolinguistic Sociolinguistic Sociolinguistic Textual

ASSIGNING SCORES TO ITEMS The first step will be to take the test the teachers themselves and then they determine the value of scores rating according to how central to the course material or how difficult they have perceived the item to be. Then scores will be given to each item taking into account that each item will be given the same weight, and each objective will be assessed with more than one item. For writing and speaking, formats will be used according to the following scale : 1-5 rating on the following: SPEAKING Overall fluency Communicative efficacy Grammar Vocabulary Pronunciation WRITING Communicative efficacy Text organization Grammar Vocabulary Spelling & Punctuation ITEM ANALYSIS Item difficulty Refers to the ratio of correct responses to total responses for the tests. A norm referenced assessment instrument, which aims to differentiate among respondents, should have items that approximately 60% to 80% of the respondents answer correctly (Cohen, 1994). Piloting of the achievement tests As the achievement tests are intended to perform an important function in an institution or in a Colombian region, they would undergo piloting on a small sample of respondents similar to those for whom it is designed. In order to do that after the teachers themselves have answered the tests they would complete an evaluation questionnaire. Students as well be asked to answer questions that give information in relationship with : How far they go when the time ran out then go ahead and complete the test so that there is feedback on all items in the test Valuable insights about what part of the instrument needs to be revised. (See evaluation questionnaire)

ADMINISTERING THE ACHIEVEMNT TESTS Cohen (1994), advises teachers to give the students a brief review of those elements that will appear on the instrument. Therefore, sample questions will be given to the students before taking the exam for the first time. Sample items usually help allay test-taking anxieties. They can serve as a warm-up to get the learner into the task at hand, a matter that should not be minimized. Poor performance may be frequently traced to inadequate warm-up. Giving instructions and a sample item or two before students receive the measure itself is advisable. The following checklist of administrative tips applies as suggestion given by Cohen (1994) The room should have adequate ventilation, heat, light, and acoustics. If a videotape is used, it should be set up and tested in advance to make sure that it works. The test administrator should assume an affable but stern posture. Smiles can help to put the respondents at ease. If the instructions are read aloud, they should be read slowly with no departure from the established wording. The time that the exam begins and the total time remaining for the test and or students subtests should be written on the chalkboard.

HOW TO EVALUATE THE TESTS

RELIABILITY The reliability of a test concerns its precision as a measuring instrument. Reliability asks whether an assessment instrument administered to the same respondents a second time would yield the same results. Three different types of factors contribute to the reliability of language assessment instruments: test factors, situational factors, and individual factors Test factors include the extent of the sampling of the objective, the degree of ambiguity of the items and restrictions on freedom for response, the clarity and explicitness of the instructions, the quality of the payout, the familiarity that the respondents have with the format, and the length of the total test. These test factors will give the test internal consistency. The reliability factor will be measured with Alpha de Cronbach, software designed to measure reliability of tests. VALIDITY Validity refers to whether the achievement tests actually measures what they purport to measure. Thus the test must be reliable to be valid. Assuming that the test is producing a reliable measure, the test must have content and construct validity. Content validity: The adequacy with which the test samples the content for objectives of the course or area being assessed. The content of each item and of the

test as a whole is evaluated. In order to find this out, the test will be evaluated by test takers and teachers. Construct validity: Whether the instrument is true reflection of the theory of the trait being measured: the language in this case. In order to show construct validity, the achievement indicators will be evaluated in different ways. That is to say one achievement will be evaluated with different item elicitation and item format. Piloting and evaluation of the instrument given by students and teachers will contribute to the validity of the achievement tests.

CUESTIONARIO PARA EVALUAR LOS EXAMENES DE RENDIMIENTO ACADEMICO DE INGLES EXAMEN CORRESPONDIENTE AL GRADO ________ NOMBRE DEL EVALUADOR: ___________________________________ GRUPO QUE RESPONDI :______________________________________ COLEGIO: ____________________________________________________ FECHA DEL EXAMEN: _________________________________________ Estimado (a) profesor (a) estudiante: Queremos conocer sus impresiones sobre el examen. Por favor responda las siguientes preguntas. La informacin suministrada por usted es de mucho valor para nosotros pues nos permitir mejorar el instrumento de evaluacin. A. SOBRE LAS INSTRUCCIONES 1. Las instrucciones para cada seccin son claras? Si___ No___ Por qu _____________________________________________ 2. Los ejemplos de cada seccin son suficientes? Si_____ No_____ Por qu? 3. El tiempo es adecuado? 4. El mtodo de administrar el examen se estableci cuidadosamente? (otra persona lo puede administrar exactamente de la misma manera?)

B. SOBRE EL CONTENIDO 1. El examen evala los logros propuestos para el grado? 2. Le parece conveniente evaluar todas las habilidades lingsticas?

De su opinin sobre la forma como se evala Habla: Escucha: Lectura: Escritura: 3. El examen evala todos los componentes de la competencia comunicativa suficientemente? Hay algn nfasis innecesario o al contrario falta enfatizar algn componente? 4. (Solo para profesores). Por favor tome el examen y califique cada pregunta de acuerdo con la importancia de ella. Escriba al lado de cada pregunta: MI: Muy Importante I: Importante M: Media 5. (Solo para profesores). Marque en el examen cada pregunta, con relacin a la dificultad de la misma. MF: Muy fcil F: Fcil D: Difcil

C. SOBRE LA FORMA DE LAS PREGUNTAS Y EL DISEO 1. Cmo le parece la extensin del examen? Muy largo______ Muy corto_____ Adecuado_____ 2. Cmo le parece la presentacin del examen? Es clara_____ Se presta a confusiones ____ Comentarios 3. Cul es su opinin sobre el puntaje dado a cada pregunta y a cada habilidad.

PREPARING THE TEST FOR SIXTH GRADE

For sixth grade the following is the list of the common objectives found in the four schools: Students ask and answer greetings and farewells Students introduce people Students give personal information Students perform some commands in the classroom Students use numbers in different situations (1 to 1000) Students give information about family members Students ask and answer about occupations and nationalities Students spell the words known. Students describe a house Students tell and understand the time.

Each school planned to teach some objectives that were not shared in the four schools: They were: Students give information using parts of the body Students give information using the days of the weeks and the months of the year Students inform about actions in progress Students give information about food Students describe pets animals Students express abilities with can Students inform about daily activities.

SAMPLE OF QUESTIONS
TEXTOS INCOMPLETOS Juan Pablo Montoya (1)______from Bogota, he was born in 1975. He started racing when he was very (2)_______. Juan Pablo won the 500 miles in Indianapolis, USA in 2000. Now, (3)________ runs in formula one with the Williams team. 1. (A) was (B) is (C) am

2. (A) young (B) intelligent (C) old 3. (A) she (B) I (C) he

DIALOGOS INCOMPLETOS A: Hello, Ben. How are you?

B: ____________________ A: Ben this is my daughter. Lisa, this is my friend, Ben. B: Nice to meet you C: Nice to meet you too (A) How are you? (B) Im fine, thanks (C) Good morning

RELACIONAR GRAFICAS CON TEXTOS Lea la carta y relacione los prrafos con las grficas que describa cada uno Picture 1. Picture 2 Picture 3

(1) Dear Jose, How are you? Im fine. Im sending these pictures of me in New York. I visited that city last week. It is a very big and exciting city. I loved Central park, it is beautiful in summer. Many people play and run there. (2) I have English classes in a Community College. Im in a class with twenty students. They come from different countries: Japan, Brazil, India, and Ecuador. (3) I live in an apartment with two Brazilian boys, we speak in English because I dont understand Portuguese, and they dont speak Spanish. They are brothers. Thy are very friendly. Write to me soon. Johan QUESTIONS FOR LISTENING COMPREHENSION 1. Look at the pictures and choose the picture that corresponds to the description you hear. 2. Listen to the words and choose the word you hear A. mother A. B brother B. C. soccer C.

3. Listen to the times you hear and choose the corresponding picture. SPEAKING

1. Answer these PERSONAL QUESTIONS Whats your name? How old are you ? Who is your English teacher? Whats your address? Whats your telephone number? Do you have any brothers or sisters? What s your fathers occupation? Whats your mothers occupation? 2. Look at these family photos and describe the people in the pictures.

REFERENCES Bachman, Lyle F. (1991). What does language testing have to offer? TESOL Quarterly, 25(4), 671-704. Bachman, Lyle F. (1990). Fundamental Considerations in Language Testing. Oxford: Oxford University Press. Burns, Anne (1999). Collaborative action research for English language teachers. Cambridge: Cambridge University Press. Cohen, Andrew. (1994) Assessing Language Ability in the classroom. Boston: Heinle & Heinle. Colombia. (1999). Ministerio de Educacional Nacional. Lineamientos Curriculares. Idiomas Extranjeros. Colombia. (1999). Ministerio de Educacin Nacional. Nuevo Examen de estado para el ingreso a la educacin superior. Cambios para el siglo XXI. Idiomas. Cmara de Comercio del Huila. (2001). El futuro Est Sealado. Corporacin Huila Futuro. Prospectiva del Huila y Formulacin de la Vision al ao 2020. Neiva: Editora Surcolombiana S.A. Delgado Santa Gadea, Kenneth. (1996). Evaluacin y Calidad en la Educacin: Nuevos aportes, procesos, y resultados. Santaf de Bogot: Coleccin Mesa Redonda Magisterio. Estvez, Solano, Cayetano (1996). Evaluacin integral pro procesos: una experiencia construida desde y en el aula. Santaf de Bogot: Mesa redonda Magisterio. Jonz, J. (1990) Another turn in the conversation.: What does cloze measure? TESOL Quarterly, 24(1), 61-83.