Fall 2012 Spec Packet

Fall 2012 EIL 460 Test Specification Packet
Intermediate Oral Grammar Virgiawan (Virgi) Kristianto, Scoot Marta, Daniel Johnson, and Julie Kierski Version 1 ----------------------------------- page 1 Version 2 ----------------------------------- page 59
Understanding Katie Honken, Version Version Version
Instructions Sophie Qin, Becky Menendez, and Mica Swyers 1 ----------------------------------- page 23 2 ----------------------------------- page 82 2.1 --------------------------------- page 99
The Barry Spec -- Testing Pragmatics Alberto Domnguez, Widya Ratna Kusumaningrum, and Yelena Forrester Knezevic Version 1 ----------------------------------- page 37 Version 2 ----------------------------------- page 116
Amending the SAT for English Language Learners (ELLs) Wahru (Esti) Sumarno, Jian Tao, and Erin Lamboi Version 1 ----------------------------------- page 46 Version 2 ----------------------------------- page 135
Intermediate Oral Grammar, Version 1 Virgiawan (Virgi) Kristianto, Scoot Marta, Daniel Johnson, and Julie Kierski
Intermediate Oral Grammar Achievement Test V1.0

Daniel Johnson Julie Kierski Virgiawan Kristianto Scoot Marta
Spec Packet, EIL 460, Fall 2012, page 1
Background: This test is designed for students in an intensive English program. It is intended for students at an intermediate level. In our program, students are placed into one of six level (100-600), and this test is an achievement test for those at the 300 level. Thus, if both instruction and learning were successful over the semester, the students should be able to fulfill all requirements of the test. The test requirements were built from the 300-level (See Appendix Figure 1) as written for the Intensive English Institute at the University of Illinois at Urbana-Champaign. Our intent in designing this test was to measure students' ability to produce grammar and grammatical structures orally as well as in writing. It is very common for oral grammar production to lag behind written grammar recognition and production. Therefore, after administering this test, we should have a better understanding of students' abilities in written grammar vis--vis oral grammar. This can then inform instruction if there is an unacceptably large gap between the two. In addition, administering this test (and the studying that students will do for it) will be an additional learning opportunity to attempt to encourage students to incorporate new grammar into their everyday usage.
Description of Assessment: This oral grammar test is one part of a final achievement-based assessment of how well students meet the proficiency goals for the 300-level course set, which consists of four separate courses (Listening, Speaking, and Pronunciation; Grammar; Reading; and Writing) each taught by separate instructors. The final assessment consists of a series of separate tests assessing each course/skill in an attempt to gather sufficient evidence to determine the students mastery of the overall course goals. The final assessment will be given once at the end of the semester. To pass on to the next level of foreign language instructions, students must demonstrate achievement of very well or satisfactorily on all four tests. A students end-of-term achievement scores in each of the areas will be combined together and used along with the teachers collective assessments to determine the students placement level for the next term. This oral grammar test will assess students ability to produce specific grammatical forms in the context of communicative tasks. The format of this oral test does not allow for the best assessment of certain discrete grammar forms and all of the benchmarks at the 300 level. Therefore, a separate written grammar test assessing those benchmarks will be given to the students; this test will also contribute to the final assessment. Since the oral grammar test items are communicative tasks, the data gathered from this test will also be used to assess the students communicative ability, specifically listening, speaking, and pronunciation. This spec document will only cover the oral grammar test section.
Description of Test Takers and Test Environment: Test takers: The students who will take this achievement task will have completed the 300-level course set at the IEI. Students generally have an intermediate level of grammar knowledge and fair communicative competence. The students are presumably familiar with the types of tasks in the test as they have encountered similar tasks in classroom activities. Environment: This test will be administered in a language laboratory equipped with an audio video system. The audio video system will be used to record students responses and to enable the evaluators to assess students responses outside the room. There will be two people inside the room, one interviewer and one student. The interviewer is the person who will give students instruction and prompts as necessary.
Objectives: Through the oral grammar component of the achievement assessment evaluation the student will demonstrate how proficient they are orally in the following grammar proficiency benchmarks for the course. Ability to ask Wh- questions accurately Relate events with clarity using a variety of verb tenses in present and past, including both simple and progressive aspects Show an awareness of countability and use appropriate nouns forms, articles, and quantifiers at least some of the time Use modals accurately to express ability, necessity, and possibility in the present and future time frames, where appropriate
General Instructions: The test will be administered by one of the students teachers (from one of the courses in the 300-level course set) so the student will be comfortable conversing with and being in a room with the administrator. The interviewer should make the student feel as comfortable as possible. The test will start with a series of warm-up questions to ease into the test. These warm-up questions are part of the assessment. Then the student will complete a series of seven tasks that focus on different grammatical aspects. These tasks will focus on one or two major grammatical forms and consist of an interview format, text or picture prompts, or scenarios. Throughout the oral test, the interviewer should give the student sufficient time, if the student needs it, to think through the task. The student may need some time to examine a picture or think
about the scenario. If the student is struggling with the task, the interviewer may try to elicit an answer by asking guiding questions. This should not be a problem though as the students will have encountered all the tasks before, so they will be familiar with the tasks formats. Some tasks require the interviewer to explain the instructions to the student. In the sample tasks below, these instructions are bold-faced.
Student Response Attributes: Student answers should be only in English and in complete sentences. As this is a test of English, any responses or utterances in the students first language will of course not be assessed. Since this is a grammar test, the content of student responses is not the assessment target and therefore will not be assessed. This is a fact that the grammar evaluator should bear in mind. In practice, however, this may be somewhat more difficult to enact because grammar and content are inextricably intertwined. Some assessment tasks require the use of a pencil and worksheet, which the student will be given. These materials are intended only for specific tasks and not for note-taking or practice.
Scoring: All of the students utterances produced within the test room setting will be open to assessment. This includes confirmation checks, clarification requests, and any asides. This will provide good data on the students natural production of English especially for the assessment of listening, speaking, and pronunciation. However, the focus of the assessment is on the language produced through the tasks, so mistakes made in these sorts of situations should have a relatively small impact on assessment. a) Two classroom foreign language teachers who will do the scoring. One of the teachers will solely focus on accurate use of the grammar orally and assign a rating to the student. Even if a student misses the prompt or context, they will only look at the spoken grammar produced, whereas the other, the LS teacher, will focus more on content, fluency, accuracy, and use of conversation and listening strategies. All of these teachers will have training and be familiar with proficiency goals to which they are grading. b) The oral test will also be videotaped. If there is disagreement among the two teachers or if one teacher is wavering in their judgment of proficiency for a student, he or she can access the recording. Then, together the two teachers can discuss an appropriate score c) For each grammar component being assessed in the tasks, students will receive a score of 1) very well, 2) satisfactorily, 3) inconsistently, or 4) unsatisfactorily. The conditions and
scoring scale are listed below in the chart at the end of this document. (See Appendix Figure 2)
SAMPLE TASKS
Oral Warm-Up Interview

Interviewer General Instructions: This warm-up interview should feel more like a friendly conversation than a formal interview. The interviewer will ask the student a few questions to engage the student in conversation. These questions will elicit students answers that use the present simple tense. The interviewer may respond to indicate that he or she is listening to the student and may ask for clarification if an utterance or meaning is unclear. However, the interviewer should not dominate the conversation. If a student is more reticent, the interviewer may have to ask more questions to elicit responses. Below are sample questions that the interviewer may ask. Easier questions should be asked at first; if the interviewer determines that the student is more advanced, then progressively more difficult questions can be asked. What do you do in your free time? Whats your favorite restaurant and why? What is your home country like? What do you like about living in America? Task Instructions to Give Students: There are no instructions for the students. This warm-up interview should be a natural conversation between the interviewer and the student. How to Formulate New Tasks: New questions to ask the student can be formulated based on the content covered throughout the courses. Questions should address topics that the student is familiar with and has gained lexical knowledge of through lessons taught. Questions should elicit answers using the present simple tense.
Task 1: WH questions
Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of WH questions. This test should last no longer than five minutes. In this task, the evaluator will 1) pose a question to the student about a certain character that he or she is very familiar with, then take that role as the students chosen character, 2) direct the student to ask WH questions with regards to his or her chosen character, 3) ask the student to create one question for practice. This practice question will not be scored, 4) answer the WH questions of the student using complete sentences instead of chunk of words, 5) prompt the student for what he or she should ask, using one word only such as hobby or address if the student consistently struggles to create questions, 6) record the response of the student, 7) end the task if the student stops producing the intended responses or the time is over. Task Instructions to Give Students: In this part of the task, imagine a character that you are familiar with and curious about such as your favorite celebrity. Ask WH questions about that character to the interviewer. The WH questions must use simple present tense. How to Formulate New Tasks: When choosing another situation, make sure the situation is widely known for triggering the students participation in asking questions using simple present tense. The chosen situation needs to be really familiar for both students and the evaluator and to focus on the use of the grammar rather than other aspects of language. Avoidance needs to be made upon situations which require too much technical vocabulary.
Task 2: Simple Past

Interviewer General Instructions: This portion of the test will focus on how well the student can correctly ask questions in the simple past. This task should last no longer than five minutes. In this task, the interviewer will hand the mock schedule (Students Sheet) to the student, then read the directions to the student. The student should be allowed 15-20 seconds to look at the schedule and think about what they want to say and then respond orally. This is an information gap activity, wherein the students job is to complete the blank portions of the schedule by asking questions. The interviewer will respond to the questions using the Interviewers Sheet. Ideally, the student will employ a variety of verbs and question words. Be as clear as possible that this what John DID on Monday and Tuesday. Examples: What time did Johns chemistry class start on Monday? What did John do from 11:00 AM to 1:00 PM on Tuesday? When did Johns soccer practice end on Monday? What class did John have at 10:00 AM on Tuesday? Task Instructions to Give Students: Look at the schedules. They show what John, a university student, did on Monday and Tuesday. There are several blanks in the schedule. Ask questions so that you can fill in all the blanks. Remember, the schedules show what John DID on Monday and Tuesday.
STUDENTS SHEET
Johns Schedule
9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-_:__ PM Monday Spanish Class Tuesday 9:00-10:00 AM 1:00-2:00 PM 2:00-_:__ PM _:__-8:30 PM Spanish Class Lunch Study
Geography Class Chemistry Class Soccer Practice Study
11:00 AM 1:00 PM
10:00-11:00 AM
__________ Class Math Class Dinner
11:30 PM-12:30 AM
_:__-11:30 PM
8:30 PM 12:00 AM 12:00-1:30 AM
INTERVIEWERS SHEET
Monday 9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-5:00 PM _:__-11:30 PM 11:00 AM 12:00 PM Spanish Class Lunch Tuesday 9:00-10:00 AM Spanish Class English Class Lunch Nap
Geography Class Chemistry Class Soccer Practice Dinner Study
11:00 AM 1:00 PM 1:00-2:00 PM 2:00-3:30 PM 8:30 PM 12:00 AM 12:00-1:30 AM 7:30-8:30 PM
10:00-11:00 AM
Math Class Dinner Study
11:30 PM-12:30 AM
Watch TV
Watch Movie
How to Formulate New Tasks: If creating a new schedule or activity, ensure that the fictitious persons name will be easy to pronounce and clearly identifiable as either female or male to avoid difficulties in pronunciation or subject pronoun choice. Choice of classes and activities in the schedule should be tailored such that they will be recognizable to students in order to retain the focus on grammar and not on vocabulary. If the day of the test is known in advance, it might be beneficial to make the schedules for the two days preceding the test. If an activity other than a schedule is chosen, use one which the student will be familiar with and will elicit various types of Wh- questions and verbs.
Task 3: Modals (Possibility)

Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of modal of degree of certainty in present tense. This test should last no longer than five minutes. In this task, the evaluator will 1) ask the student to imagine a certain situation in which he or she can make guesses such as guessing gifts, the possibility of a certain team winning a game, or weather prediction, 2) have the student create 5 guesses to which you must not respond, 3) forbid the students to jot down his or her guesses before he or she produces them orally, 4) prompt the student if he or she pauses for certain amount of time, 5) end the task if he or she struggles to construct the guess or the time is over. Task Instructions to Give Students: In this part of the task, construct 5 sentences using modals of degree of certainty in present tense. In this part of the task, you will 1) imagine your birthday party 2) guess orally 5 gifts that you get using modals of degree of certainty in simple present tense How to Formulate New Tasks: Other scenario might be applicable for this task. However, it is necessary to set a scenario in which people usually make guesses such as guessing gifts, a weather report, a football teams possibility to win a game, etc. Have the student choose one of the scenarios that he or she really understands.
Task 4: Present Progressive

Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use the present progressive and control it in conversation. This task should last no longer than five minutes. In this task, the evaluator will hand a picture to the student, then read the description. While listening closely to the description, the student should look at the picture. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. This is an adaptive test. The test will begin with the student speaking freely, but if a student consistently struggles to answer, the interviewer can help direct the students focus by pointing to a picture and ask a WH- question. For example, What is this lady doing? What is the shoe store owner doing? Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture, the interviewer can end the test. Task Instructions to Give Students: Look at the picture below. Imagine that you are in your apartment. You are looking out your window and this is what you see. Describe all the things that you see that are happening. You have 15 seconds to review the picture before you speak.
(Communicative Grammar Practice, Leo Jones, 1985)
How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card. This would create a more challenging task for students though since the students wont have the 15 seconds to compose their students as in the picture task. Additionally, the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. When choosing another picture, its important to make sure that the picture has enough images to produce a wide variety of sentences in the progressive form. Whatever image is used, its important that the students have general knowledge of the vocabulary in order to retain the focus on grammar, otherwise it could become a test of vocabulary knowledge.

Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use past progressive and control it in conversation. This task should last no longer than five minutes. In this task, the interviewer will hand the picture over to the student then read the description. While listening closely to the description, the student should look at the picture. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. The test will begin with the student speaking freely, but if a student consistently struggles to answer, the interviewer can help direct the students focus by pointing to a picture and ask a W-H question. For example, What was this man doing? Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture, the interviewer can end the task. Task Instructions to Give Students: Look at the picture below. Mr. and Mrs. Gold invited some friends to their house for the weekend. A thief stole Mrs. Golds jewelry at midnight. Describe all the things that the guests were doing when the thief stole the jewelry at midnight. You have 15 seconds to review the picture before you speak.
(Azar Beginner Grammar Textbook Series)
How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card. This would create a more challenging task for students though since the students wont have the 15 seconds to compose their students as in the picture task. Additionally, the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. When choosing another picture, its important to make sure that the picture has enough images to produce a wide variety of sentences in the progressive form. Whatever image is used, its important that the students have general knowledge of the vocabulary in order to retain the focus on grammar, otherwise it could become a test of vocabulary knowledge. If names are assigned to people in the picture, as in this sample task, make sure the names of the fictitious characters can be pronounced easily like this task does with colors.
This task can also be made into a 1-on-1 conversation by asking the student questions about what they did yesterday during certain times. For example, the teacher could imagine that they are a policeman and interview the suspect (the student) by asking a variety of questions that should product the progressive aspect such as What were you doing at 8 AM yesterday? or We found evidence of food in your car. What where you eating in your car?
Task 6: Modals (Necessity)

Interviewer General Instructions: This portion of the test will focus on the students knowledge and control of necessity modals. This test should last no longer than five minutes. In this task, the evaluator will read the prompt to the student. The student will have 15-20 seconds to think before beginning to respond orally. If the student seems to be having serious difficulties with inventing rules (i.e. generating content), some prompting may be appropriate, such as What about homework? or What about talking in class? Care should be taken not to suggest too many forms to the student, however. Task Instructions to Give Student: Imagine you are a teacher in a high school. It is the first day of class. Think of some rules that you will tell your class on the first day. How to Formulate New Tasks: Other scenarios are, of course, imaginable for this language feature. Whatever might be chosen, ensure that it is possible to elicit various necessity modals, bearing in mind that avoidance, whether intentional or unintentional, may intrude. Try to structure the task in such a way that it is not possible (or not easy) to use ability modals like can and cant. If students are not comfortable with necessity modals, they may try to avoid them altogether by using ability or advice modals. Both activity and directions can be useful tools to attempt to prevent this sort of avoidance.
Task 7: Articles / Quantifiers

Interviewer General Instructions: In this task, the student will be given a photo of an open refrigerator/freezer with various kinds of food inside of it. The student will describe the items in the photo. The student will be assessed on the correct use of articles and quantifiers with the accompanying count and non-count noun(s). This task will also produce data for the assessment of verb agreement in the student's utterances. The student may be creative in their descriptions and produce longer sentences, but only the articles, quantifiers, and verb agreement will be assessed for the oral grammar assessment. This task should take no longer than five minutes. The student should describe at least 6 items in the photo. If the student only produces statements using count nouns, the interviewer should prompt the student to describe a non-count item, and vice versa. If the student only describes plural nouns, then the interviewer should prompt the student to describe a single item (to elicit the use of the articles "a" or "an"), and vice versa. If the student is having trouble producing the sentences, the interviewer may elicit responses by using guiding statements or questions such as the ones below: Interviewer (pointing to an item): Can you describe this? Or if the student needs more guidance, Interviewer: Are there any apples in the fridge? Student: Yes, there are. Interviewer Follow-up: How many apples are there? Student: There are a lot of apples in the fridge. Task Instructions to Give Students: Describe what you see in this picture. You may use words such as "a lot of", "lots of", "many", "some", "a", and "an".
How to Formulate New Tasks: Other picture scenarios, such as a shelf in a store or a room in a house, may be used for this task. Whatever picture that is used should show a variety of count and non-count nouns; these nouns should be ones that the students should have in their lexical knowledge through lessons that were taught in the 300 level or in the previous 100 or 200 level courses. Pictures can be black and white line drawings, color drawings, or color photos as long as the picture is clear and large enough to see the individual items in the picture.
Appendix
Figure 1
Figure 2
SCORE Grammar
A few errors, overall communication of ideas is clear
Very Well
Satisfactory
Several errors, overall communication of ideas is still clear
Inconsistently
Significant and consistent errors that sometimes confuse meaning
Warm-up
Simple Present WH- Questions
Many errors that making meaning unclear. Listeners attention is diverted to the errors from the message.
Unsatisfactorily
Task 1: Task 2:
Simple Past Possibility Modals
Task 3:
Present Progressive
Task 4:
Task 5: Past Progressive Task 6: Necessity Modals Task 7: Articles
Quantifiers
Understanding Instructions, Version 1 Katie Honken, Sophie Qin, Becky Menendez, and Mica Swyers
Introduction
This test is designed to serve as a supplement to the English Proficiency Test (EPT) given at the University of Illinois Urbana-Champaign for incoming non-native speakers of English. Assuming incoming students have a sufficient score to enter the university, this test targets intermediate-level English-language learners who would be expected to succeed in American university-level courses. While the EPT tests writing skills and pronunciation, this supplement will assess students ability to succeed in a university course and classroom setting by following instructions in a variety of formats. Students who do not demonstrate sufficient ability will be placed in a basic classroom survival skills class in conjunction with their ESL courses at the University of Illinois. This EPT supplementary section is necessary to reduce the influence of students ability to follow instructions on their performance in a university-level course (i.e., losing points due to inability to follow instructions properly).
General Objectives
The objective of this test is to measure students ability to understand and implement different forms of feedback in an academic setting. Additionally, it will identify which topics need to be covered in a classroom survival skills course.
Specific Objectives
1. 2. 3. 4. Assessing students ability to understand written directions of assignment. Assessing students ability to understand and follow oral directions for in-class work. Assessing students ability to understand and apply a rubric. Assessing students ability to understand written feedback.
Objective #1: Assessing students ability to understand written directions of assignment

Item 1: Formatting
Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in a written format: detailed instructions of editing a paragraph will be provided to examinees, and those instructions are required to be understood by students in university-level courses. The examinees will face four types of formats in screenshots, among them only one format is correct and the other three are with different problems including spacing, margin, size or editing mistakes in the title. The key screenshot correctly follows the instructions provided in the item stem. The three distractors will depict various formats that arise from misunderstanding of the instructions. Response Attributes (RA) To select the right format from the four options, the examinees need to read through the instructions. Then they will the information from instructions to the formats displayed in screenshots, and select the format that meets all requirements. The computer testing software will record the result and categorize it as correct or incorrect. Correct answer earns 2 points while incorrect answers earn 0 points. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. C: The title is not bold and the text size is 10pt. D: The font is not Times New Roman. Sample Question: In a writing class students are required to edit the provided passage based on the following format: Font: Times New Roman (This is what Times New Roman looks like.) Spacing: Double-spaced Size: 12pt (This is what 12pt. font looks like.) Title: Times New Roman, bold, centered Which of the following formats meets the teachers requirement? ________
Choice A
Choice B
Choice C
Choice D
Item 2: Writing correct file names

Prompt Attributes (PA) This is a writing test item. The item stem will be given in a written format: detailed instructions of writing a correct file name will be provided to examinees, and a sample file name is also presented to help them understand the instructions. The instructions of writing correct file names are required to be comprehended by students in university-level courses. The right answer should be correct in all aspects including the sequence of various information, capitalization and symbols. Response Attributes (RA) Examinees will come up with correct file names based on the information provided. This may require examinees to read through the instructions and understand the sample presented. Correct answer of a whole file name earns 5 points, and each incorrect portion deducts 0.5 point. Sample Question In a writing class, the teacher requires students to upload their homework on the course management system with correct file names. The file name should follow this format: mm_dd_Last Name_First Name_Title_of_Paper Example: If it is April 15th, and Robert Pattinsons paper is named Environment Issues, the file name should be: 04_15_Pattinson_Robert_Environment Issues Please use the provided information and write the correct file names on the blank. 1) Students Name: Barack Obama Date: August 5th Title of Paper: How to be a Good President ___________________________________________________________________ 2) Students Name: Mitt Romney Date: November 24th Title of Paper: If I were the President of the United States ___________________________________________________________________
Answer Key:
1) 08_05_Obama_Barack_How to be a Good President 2) 11_24_Rommey_Mitt_If I were the President of the United States
Objective #2: Assessing students ability to understand and follow oral directions for in-class work.
Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in oral format: the examinee will listen to a sound file of a teacher giving instructions for an in-class assignment. These will be instructions that a student would hear in a university-level course, for tasks such as groupwork, peer review, or an in-class assignment. The examinee faces a computer monitor divided into quadrants (one for each choice), each with a different illustration. The key picture correctly illustrates the directions given in the oral item stem. The three distractors will depict different scenarios that could arise from following the instructions incorrectly or incompletely. Response Attributes (RA) To select the correct choice of the four options, the examinee will have to listen to the sound file. Then s/he will have to process the information and select the picture (by clicking) that best illustrates the instructions that were given. The computer testing software will record the item that the student selected and categorize it as correct or incorrect. Sample Item The examinee(s) should hear: Now, get into groups of three or four and come up with a paragraph responding to the question on the board. You should only turn in one paragraph per group at the end of the class.
(Answer Key to Sample Item: The correct choice is Picture A, where the figures are discussing and producing one paper. Picture B is incorrect because each student is turning in a separate paper, and the first students paper is multiple pages long. Picture C is incorrect because each student is working separately (and without discussion) on a paragraph. Picture D is incorrect because while the students are engaged in a lively discussion, there is no evidence that they are producing a paragraph to submit at the end of class.)
Objective #3: Assessing students ability to understand and apply a rubric.

Prompt Attributes (PA) This is a four-item, multiple-choice test item. Each item consists of a stem and two questions. The stem briefly introduces a fictional student who must complete an assignment. It also introduces a rubric (in the form of a table) for that assignment. Next, it introduces the fictional students attempt at the assignment (in the form of a screenshot). Of the two questions, the first question focuses on one of the rubric categories and asks how many points the fictional student earned with their attempt at the assignment. The choices are always 3, 2, 1, and 0. The only correct choice is whichever number corresponds to the score the fictional student should receive in that specific category, based on the requirements laid out by the rubric. The second question asks students to identify one change the fictional student could make to increase their score in the specified category. One choice will always be nothing because the fictional student has already earned the maximum score in that category. The other choices will each reflect one of the more specific requirements of the assignment. In cases where the fictional student has earned the maximum number of points in the specified rubric category, the key will be the choice (A) that states that the example student should change nothing. The first distracter (B) is incorrect because it recommends a change for the wrong category. The second and third distracters (C & D) are incorrect because they recommend changes to components that are already correct. In cases where the fictional student has NOT earned the maximum number of points in the specified rubric category, the key will be the choice (D) that correctly identifies one change that would raise their score in the specified category. The first distracter (A) is incorrect because the fictional student did not earn the maximum number of points. The second distracter (B) is incorrect because it is recommending a change for the wrong category. The third distracter (C) recommends the wrong change, although for the right category. The student should not recognize the key immediately. They must examine both the rubric and the assignment, and they will compare the assignment against the rubric requirements to acquire the information they need to correctly answer the question. The grammar and complexity of the scenario description and item directions should be very basic. Students should not feel challenged by the language of the prompt itself. The grammar and complexity of the assignment should not exceed that of the materials they might be expected to encounter in ESL114 class materials. In general, they should also be fairly basic, so that reading comprehension abilities do not interfere with the students ability to identify the correct answer. Rubric categories should not include technical writing terms or concepts (as students may not have received instruction concerning these yet). However, rubrics should
include language very comparable to what is used in ESL114 rubrics, even if this language is more challenging than the language contained in the prompt or assignment. Response Attributes (RA) Students will select the correct answer for each of the two questions in the item. This may require re-reading of the rubric and of the assignment screenshot. Students will be assessed on whether each of their selections is correct or incorrect. Correct answers earn 1 point each. Incorrect answers earn 0 points. Thus, each of these types of items may earn a maximum of 2 points (1 for each of the two questions). Sample Item: Jemima has been given a paragraph writing assignment in her ESL class. The instructor provided the following rubric, by which the paragraph will be scored: Paragraph Rubric 3 points Topic sentence 2 points 1 points
The main idea is unclear.
0 points
The paragraph does not have a main idea.
The main idea of the The main idea is paragraph is clear and clear but it is incorrectly placed. appears at the beginning of the paragraph. Paragraph has at least three supporting detail sentences that relate back to the main idea. Grammar errors and incorrect word choices do not distract the reader from the important points Paragraph has two supporting detail sentences that relate back to the main idea . Grammar errors and incorrect word choices are distracting.
Support
Paragraph has only one supporting detail sentence that relates back to the main idea. Grammar errors and incorrect word choices make it difficult to understand the paragraph. The paragraph meets 1 out of 3 of the formatting requirements.
Paragraph has no supporting detail sentences that relate back to the main idea. Grammar errors and incorrect word choices prevent the reader from understanding the paragraph. The paragraph does not meet any of the 4 formatting requirements. The paragraph is less than 65 words long.
Grammar
Formatting The paragraph meets The paragraph

all 3 formatting meets 2 out of 3 of requirements: double- the formatting spaced, Times New requirements. Roman, Size 12.
Length
The paragraph is at least 85 words long.
The paragraph is 75 The paragraph is 65 to 84 words long. to 74 words long.
When Jemima finishes writing her paragraph, this is what it looks like on her computer screen:
Question #1: Based on the instructors rubric, how many points will Jemima get in the category of formatting? A. 3 B. 2 C. 1 D. 0 Question #2: What is one change Jemima could make to get more points in the category of formatting? A. Nothing. She already has the highest number of points possible in this category. B. She should make her paragraph longer. C. She should change her paragraph spacing. D. She should change her font size.
Key: 1. B 2. D Adapted from: Writing Academic English. Alice Oshima and Ann Hogue. Fourth Edition. The Longman Academic Writing Series
Objective #4: Assessing students ability to understand written feedback

Prompt Attributes (PA): This is a four-option multiple-choice test question. The task tests test-takers ability to understand and apply written feedback. Test-takers are shown a screen-shot of a paragraph written by a student with teacher comments. Four options will be shown as screen-shots of the same paragraph with changes. Three of the options will show the paragraph with incorrect or incomplete application of teacher feedback. One option will show one way that the teacher feedback could be applied well. More than one option might seem to apply all of the teacher feedback, but test-takers should choose the best option. Response Attributes (RA): The test-taker should first examine the paragraph with teacher comments. Next, the test-taker will examine the four options and choose the one that best applies teacher feedback. Students will give either a correct or incorrect response. Sample Item The following paragraph was submitted by a student. The teacher has given the student feedback so that the student may revise the paragraph. Teacher Feedback:
Which of the following best applies the teacher feedback for the paragraph shown above? Option A:
Option B:
Option C:
Option D:
Answer Key to Sample Item: Option A: (Does not have at least five sentences) Option B: (Correct) Option C: (Attempted to apply all feedback but still incorrect) Option D: (Does not have a transition and at least five sentences, sentence split incorrectly)
The Barry Spec -- Testing Pragmatics, Version 1 Alberto Domnguez, Widya Ratna Kusumaningrum, and Yelena Forrester Knezevic
Pragmatic Test Specs: The Barry Item EIL 460, Fall 2012
Alberto Domnguez Widya Ratna Kusumaningrum Yelena Forrester Knezevic
Item types:
- Requesting information - Excusing yourself - Expressing concern Methods: - Oral interview
- Making small talk
- Short answer
- Multiple choice These specs are applicable in a classroom setting where students are aware of the pragmatic content and have agreed to be tested on potentially sensitive subjects. The test-taker will have signed off on something like the following: General specs:
- Conversation
This class will include tasks that test my ability to conduct small talk, request and give information, accept and decline invitations, excuse myself from a situation, and express concern to friends about their potentially harmful behavior. I understand that some of these topics may be sensitive. I understand that I will not be tested on whether my answers follow strict patterns or formulas, nor on whether I agree with the implications of certain test items. Rather, I will be evaluated based on the politeness and appropriateness of my responses. General Objectives: Test takers will use their rationale to demonstrate integrated competences (strategic, discourse, and sociolinguistic). Their responses will be assessed based on adequacy and appropriateness of the language context.
Excusing Expression
1. Prompt Attribute
Test item consists of an oral and recorded instruction which asks test takers to respond using their sociolinguistics and strategic competences on the assigned topics. This item involves students understanding about interpersonal relationships (i.e. friendship, neighborhood). 1. Response Attribute Test takers will respond to the test item orally. Their utterances will be recorded using recording or other devices. Their answers will be divided into three levels (lower, intermediate, or high). These level classification are based on the use of magic formula (greeting, appropriateness level). 1. Sample Imagine that you are living in a suburb and have a good relationship with your neighbors. You usually chit-chat and gossip with them about anything. Last night you heard a police siren in the early morning. When you start driving your car, your friends come to stop your car and small talk with you. However, you dont have enough time since youre in a hurry to catch your morning flight or youre late. How will you excuse yourself from conversation politely?
2. Prompt Attribute
The test item is an oral instruction which have been recorded before. The item requires multiple competences (strategic, sociolinguistic and vocabulary). The item assesses students understanding of excusing expression which involves interpersonal relationship and academic settings (friendship between two students in schools). 2. Response Attribute Their oral responses will be recorded and assessed based on the level of appropriateness to the context (level of formality). Students will be challenged to use proper vocabulary and strategies. There are four scoring rubrics used (less formal, average formal, formal, and more formal). In this sample question, responses will be assessed according to a less formal rubric because the interlocutors are friends. 2. Sample It is a lunch break and your professor wants to meet with you. While you are walking to his office, you meet your friend from another class. He steps closer to you to talk about the tailgate this weekend. Since meeting with the professor is really important for your grade, you dont want to come late. What will you say to your friend?
3. Prompt Attribute
The item is presented as a multiple choice item which assesses students mastery of sociolinguistic politeness in excusing themselves. This item involves the concept of greeting and excusing phrases. Choice (a) and (b) will be both be incorrect choices. If we break these sentences apart, we find that the sentences have more greetings than the excusing expression itself. These sentences tend to open a conversation rather than excuse the test-taker from it. Choice (c) is the correct answer since this sentence consists of greeting phrase and excusing expression in which the tendency of the answer is still in accordance to the objective of the item. Choice (d) is a rude choice, but it becomes a good distractor as opposed to choice (a) and (b) in which students who do not understand the item so much, will choose this option. 3. Response Attribute
Students will select the correct answer based on the sociolinguistic situation in target language. Students need to master the concept of less greeting and more excusing. When excusing themselves, it is important for students to avoid too many greetings because that would lead to further conversation. After having class, you have an appointment with your doctor. You only have 1 hour to get there by bus. Since it is not easy to have an appointment with her, it is a priority. When you are ready to leave the class, your professor is calling you and wants to discuss about your study. What will you say to your professor to excuse yourself? (b) Yes Professor, theoretically test spec is not easy to understand especially how to make the proper prompt attribute and response attribute. Thats why I need to study hard about it. (d) Yes Professor, I am sorry, but I need to meet doctor right now. (a) Yes Professor, theoretically test spec is not easy to understand especially how to make the proper prompt attribute and response attribute. As a teacher candidate in non native speaking country, I realize that it is important for me to expose my students to the typical items. 3. Sample
(c) Yes Professor, I will study hard about test specs, but I am so sorry, I need to go now to meet doctor after the class. I dont want to miss it.
Requesting Information
1.Prompt Attribute: This item assesses ability to request information. The prompt will consist of a paragraph. The paragraph will indicate the relationship, or lack thereof, between test-taker and interlocutor. Then there will be another sentence which will establish the need the test-taker has and how the interlocutor is involved in the requesting of the information. The relationship will inform how the information should be requested: level of politeness, proper greeting, etc. The testtaker will then be prompted to orally request the information needed for the situation described in the prompt. 1.Response Attribute: The item will be responded to orally, simulating how the test-taker would request information. The response will include a greeting. After this, there will be a question worded appropriately based on the interlocutor being addressed. The response should avoid rudeness
2.Prompt Attribute: This item will test the test-takers ability to give information to others. The prompt will consist of a paragraph describing an interaction between the test-taker and another person. Alternately, the prompt may be presented as a dialogue leading up to the target interaction. The interlocutor in the item will vary in age and familiarity with the test-taker, and the variation will be stated in the prompt. The setting of the interaction will influence the formality of the prompt conversation. A topic of interest will be brought up in the prompt. The interlocutor will then request information from the test-taker relating to this interest. The request will come in the form of a question. The knowledge of the test-taker will vary, the prompt may tell the test-taker that she or he does not know the answer and that they must respond to the question by stating this in a manner appropriate for continuing the conversation.
1.Sample: You have gotten off at the wrong bus stop and are late to a dinner party at your friends home. You have never been in the neighborhood where you are currently lost. As you enter a gas station you see a middle-aged woman at the counter and approach her. Ask the woman for directions to your friends address in an appropriate and polite way.
while being concise, appropriate and effective in conveying meaning.
2.Response Attribute: The item will be responded to orally. The test-taker may respond by stating a lack of knowledge of the information in an appropriate manner if indicated in the prompt. If this is not stated in the prompt the test-taker will provide the information requested of them politely and concisely. The response must logically address the interlocutor and fully answer the question. A full answer will consist of a response which provides the interlocutor with information they can use to address their concern. 2.Sample: You are drinking coffee at your friends apartment. Her friend overhears you mention an upcoming concert at a local bar. He asks you whether the tickets are already for sale and if not when they will be on sale. You do not know if the tickets are being sold or when they will begin to be sold. Convey this to him and propose some other way he may find the information.
1. Prompt Attribute (based on the Barry item from LTA C3.3): This conversational task will propose a situation in which a test-takers friend is engaging in potentially harmful behavior. The test-taker will be asked to express concern to this friend.
Expressing concern
(The type of behavior can be modified according to severity, rarity, and social stigma. Overly sensitive topics should be avoided unless previously agreed to by the test-taker. However, the type should be one that people might plausibly engage in repeatedly despite its detrimental effects. Various types of addiction could fit here.) 1. Response Attribute: The test-taker will be expected to respond to the task given in the prompt. They should express concern about their friends potentially damaging behavior. A response should proceed from the established social relationship that was mentioned in the prompt. As such, the test-taker might refer to previous experiences that they and their friend have shared. They might imply that the two of them have had similar discussions in the past, and might express dismay that their friend has fallen back into those harmful old habits.
The test item will be a short written passage that establishes the social background of the two friends. First, it will be established that they were close in earlier years and still maintain friendly social contact. Next, the potentially harmful behavior will be introduced as one in which both friends used to partake. Then it will be established that the test-taker gave up this behavior while the other did not. There will be a recent example of the test-takers behavior that moves the student to worry about the friends conduct. Finally, the test-taker will be asked to raise these concerns with the friend.
The response should make sense. As long as the point is made, grading should not focus on grammatical accuracy, but on whether or not the test-taker states their concern effectively. The examiner should supply the friends answers based on the test-takers own utterances. 1. Sample: You are midway through your junior year of college. You have a roommate, Barry, with whom you have been living since freshman year. Barry is a close friend. On weekends you go out with a group of friends to movies, concerts, and parties. You enjoy talking together and frequently have long, philosophical conversations.
In the past, both you and Barry have had problems with procrastinating. You often avoided starting assignments until the last possible date, and then you typically produced poorquality work due to lack of time. The stress of urgent deadlines weighed heavily on your
minds. During final exams, the two of you would stay up late cramming for each test because you hadn't prepared well enough earlier.
You have now managed to get your procrastination under control. These days, you normally finish tasks a day or two ahead of time. Your grades have improved, and you feel less anxious. Barry also seemed to improve for a while, but now he appears to be backsliding. He spends most of his time playing video games and hardly ever opens a textbook. Last semester he just barely achieved a C average. Now, at the beginning of the spring term, he shows no sign of wanting to change. You are worried. Initiate a conversation with Barry in which you discuss his procrastination problem.
1. Prompt Attribute: This multiple-choice item will test sociolinguistic competence by assessing the test-takers response to a situation involving small talk. The stem will introduce a situation which the test-taker can be expected to be familiar with, and which would be an appropriate setting for strangers to converse. The stem will provide an initial opening of small talk by a stranger, and will ask the test-taker to choose the most appropriate response from among four choices. 1. Response Attribute: There will be four answers to choose from. They will be of the following types: a + d: Imply that the strangers comment is true, but offer a response that is minimizing and impolite. b: Ask for clarification even though no clarification should be necessary. c: Offer noncommittal agreement without minimizing or escalating the situation.
Small talk:
The test-taker should choose (c) in order to receive credit. The order of these answer types should be varied, without too many of this overall item type appearing on the same test. 1. Sample:
You are waiting in line at the grocery store, and the line is moving very slowly. The person next to you turns to you and says, Wow, looks like we picked the wrong line. Choose the most appropriate response. *c. Yeah, looks like it. d. Im not worried. a. Come on, its not a big deal. b. What do you mean?
2. Prompt Attribute: The examiner and test-taker will roleplay a situation in which small talk with strangers could be likely to occur. The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before bowing out. Then the examiner will initiate the conversation. (This task could come at the beginning of an oral interview with the examiner.) 2. Response Attribute: Each participants responses should proceed logically from the previous utterance. The test-taker will respond politely and make a few generic comments of their own. The conversation should not stray outside the boundaries of typical small talk (sports, weather, some mutual obstacle, etc.). If a test-taker is not well-versed in a topic (e.g. sports), they should be able to go along anyway, either assenting and asking questions or admitting lack of familiarity. The testtaker should not ask questions about the examiners appearance or personal life. When it comes time to bow out, the test-taker will do so gracefully, with politeness but without overfamiliarity. 2. Sample starting lines: - Wow, this rain is really something. - How bout them Steelers?
- Man, I cant wait to get out of here (or: get home).
- That house already has Valentines Day decorations up. It gets earlier and earlier every year.
- Can you believe this? The line hasnt moved in ten minutes.
3. Prompt Attribute: The examiner and test-taker will roleplay a situation in which the test-taker is responding to small talk initiated by a friend. The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before bowing out. Then the examiner will initiate the conversation. (This task could come at the beginning of an oral interview with the examiner.)
3. Response Attribute: Each participants responses should proceed logically from the previous utterance. The test-taker will respond politely and familiarly, inquiring as to the friends own opinions. The conversation should not stray outside the boundaries of typical friendly small talk (sports, weather, friends or family, some mutual obstacle, etc.). If a test-taker is not well-versed in a topic (e.g. sports), they should be able to go along anyway, either assenting and asking questions or admitting lack of familiarity. When it comes time to bow out, the test-taker will do so familiarly, using a Good seeing you or See you around type of closing statement. 3. Sample starting lines: - Hey, man, hows it going?
- Can you believe this? She said she was going to give back our grades in a week, and its been a week and a half. - Hi! I didnt see your brothers at the game last week. How are they doing?
- Hey, I havent seen you around much lately. How have you been?
Amending the SAT for English Language Learners (ELLs), Version 1 Wahru (Esti) Sumarno, Jian Tao, and Erin Lamboi
universities and colleges. Over the past 75 years, the SAT has undergone many changes, Not only in respects to test design but more so in the demographics of the test takers. However, surprisingly enough, modifications in the SAT have not reflected these particular changes. This test is Known cultural bias towards the white and the wealthy. With a growing number of English language for the oarsmanregatta analogy question, which demonstrates why the test is has been accused of the various issues involved with such a heterogeneous group, with their wide ranging ethnicities, cultural backgrounds, and native languages, becomes increasingly apparent. This issue becomes even more pressing when validity and reliability are taken into account. Background: The Changing Population of SAT Takers
The SAT is a standardized test used in determining college admission into United States
SAT Amendment for ELLs
learners (ELLs)coming from different parts of the world to taking take the SAT, the need to address
account. has to tackle. The steepest increases in test volume since 1973 have been among students of Asian or Hispanic/Latino descent; the proportion of African American test-takers has also increased. Both groups together made-up one fifth of all SAT test-takers in 2001. However, the proportion of white test-takers decreased significantly from 87 percent in 1973 to 66 percent in have had serious implications for test makers who share different first languages, cultures and
resulting in more diversified racial/ethnic backgrounds that the test makers have to take into
As illustrated in the chart below, dramatic changes have occurred to the makeup of the test takers
2001 (Lawrence et al, 2002). Over the years, these changes in the ethnic backgrounds of test takers basic knowledge of the world. In upholding the views of the validity theory, which should be the goal of not only SAT test makers but all test makers alike, there should be a focus on maintaining a
certain degree of flexibility, which will allow for modifications to be made in response to the changes in our society. However, this is not the case.
Studying the history of the SAT, how it has evolved, why it has evolved, and to what extent it has
A Brief Review of Major Changes in the SAT
evolved or been modified constitutes a great resource for future SAT test makers to use in order to
create a more reliable, more valid exam for the future. One of the major changes in the SAT has been that of time; the time allotted to each particular section of the test has gradually increased over the should not expect to finish. Even so, many of the early modifications to the test aimed at providing more liberal time limits. In 1928 the test was reduced to seven subtests administered in 115 minutes, and in 1929, to six subtests" (College Board). Then, more recently, the SAT has been modified even further to five rather than six questions; the antonym format was changed to the test more reliable, and ensure that what the test was created to evaluate was actually what was years. "Early versions of the SAT were quite speededas late as 1943, students were told that they
more familiar five-choice question (College Board)." These modifications were meant to make the
being tested. Despite the changes that have been made over the years, there is still much more work
that needs to be done. More specifically, the needs of ELLs who are taking the test in the medium of their second or foreign languages need to be addressed and taken into account when creating the SAT in order to validly administer a fair, unbiased, and reliable test. To narrow down our scope, this project will focus on sentence completion in the critical reading section, the most challenging
part of the test.
Spec of Sentence Completion in the Current SAT
1. General Description:
The SAT is a standardized test used by colleges and universities in the United States to assist in the The purpose of the test is to assess the critical thinking skills youll need for academic success in collegeand how well you apply what youve learned in school to analyze and solve problems, the way
youll need to in college (SAT official guide, pp.3). It includes critical reading, mathematics and writing sections.
The critical reading section includes reading passages and sentence completion. The writing section includes a short essay and multiple-choice questions on identifying errors and improving grammar and usage. The mathematics section includes questions on arithmetic operations, algebra, geometry, statistics and probability. The Critical Reading (formerly Verbal) section of the SAT is made up of three scored sections: two 25-minute sections and one 20-minute section, with varying types of questions, including sentence completions and questions about short and long reading passages. Critical Reading sections normally begin with 5 to 8 sentence completion questions; the remainder of the questions is focused on the reading passages. Sentence completions generally test the
student's vocabulary and understanding of sentence structure and organization by requiring the student to select one or two words that best complete a given sentence. Questions on the SAT reading-sentence completion are all multiple-choice questions which have five answer choices, only one of which is correct. The questions are weighted equally. For each correct answer, one raw point is added. For each incorrect answer onefourth of a point is deducted.
2. Prompt Attributes Sentence completion is administered by means of multiple-choice questions.. Test takers are given an incomplete sentence, with one to two missing words, that they must complete using the choices provided. The objective is To select the best choice among the five possible answers, they have to do close reading and grasp a clear understanding of the sentence structure, content and basic background knowledge associated etc.
3. response Attributes The test takers will be required to read and compare the five alternative choices presented in the test item, and select best completes the sentence, both in meaning and in form.
Modifications for ESL test takers)
1). Language Issue: The recondite vocabulary presented in the current test items prevent the test from assessing what the SAT is supposed to test. As mentioned above, the test is designed to evaluate students critical thinking and problem solving skills. Yet, in reality, the daunting vocabulary redirects the test making it more of a language test than an assessment of critical
thinking and problem solving skills. As the number of ESL learners taking the SAT increases so does the responsibility of to test makers to create a test that actually assesses critical thinking and problem solving skills rather than a language test consisting of "vocabulary-loaded" questions which render huge obstacles for ESL test takers and do not fairly nor appropriately demonstrate how well they have truly mastered the target skills. 2). Cultural Issue: The prevalence of cultural bias has been(dont need this) a consistent problem or consistently problematic in regards to the content of the test questions. Over the past decades, the major accusations against the test makers and the test itself have been related to cultural bias, specifically,towards wealthy, Caucasian Americans. the best example of this is the famous oarsmanregatta analogy question (Wikipedia). Considering the fact that nowadays the test is going moving beyond the "united states border" and targeting students all around the world, it seems that the next appropriate step, in regards to the SAT, would be to adjust to these changes. its The SAT's almost almost exclusive emphasis on western culture, politics and history tends to places certain test takers at an unfair advantage over others, 3). Time Issue: It is commonly acknowledged that we need more time to process information in the L2 than than the L1, especially when both the two languages vary so greatly as does English and non-alphabet languages (Asian languages).Along with the fact that Asian students are among the largest group of international SAT test takers, it is obvious that certain modifications need to be made in order to meet the needs of these "new" test takers, in order to ensure that what the SAT aims to test or the objectives of the test are being adequately and appropriately met we can take out this last part of the quote (College Board). Reforms to the test are imperative if validity is to be upheld.
Amendment to the Current SAT Spec: General Description
The revised version of the SAT for ESL learners, will continue to be a standardized test used by colleges and universities in the United States to assist in the admissions process. The purpose and or objectives will remain the same as well which are to "assess critical thinking skills" of high school students entering into college through testing math, writing, and critical reading skills. One mode of assessing critical reading skills, in specific, is through sentence completion, which is the focus of this spec. In regards to cultural bias, the revised SAT examples will represent various parts of the world, cultures, ethnicities, religions, etc. Just as England and the United States and their government, history, and way of life are used as examples in the SAT so will other countries be used, such as China, Indonesia, Korea, etc. In this way, certain test takers will not have an advantage over others due to their cultural background, which influences the level of familiarity a student may have with the material presented.
Along with important changes such as including questions that represent a greater range of countries and cultures, is the necessity to make the SAT more accessible. The majority of international SAT test takers must travel to the major cities of their countries to take the test, and this is if the SAT is offered in that particular area. This process of long-distance traveling not only costs money but takes time and energy, creates added stress and uneasiness, that native
speakers or students residing in the United States do not have to endure to the same extent. Therefore, making the Revised SAT available in more countries and more accessible within each of these countries by offering the test outside of the cities as well as within, more students will have access. Hopefully creating an equal opportunity for students to take the test regardless of where they live or their economic status. Just as only offering computer-based tests, with no regard to those who may not be able to afford or obtain this means of communication, shuts off certain groups of students, so does only offering the SAT in cities difficult and costly to reach. If the location of where the SAT is offered, in which country or city, is uncontrollable, then the student's specific situation should be taken into account, their time and money spent traveling as well as the difficulties faced finding room and board, location of test taking center, understanding requirements, rules of the test, coming prepared with the correct testing materials, etc. All of these factors influence the performance of the ESL student therefore affecting the results of the test as a whole. With respect to the actual test format, the sentence completion questions of the revised version will be similar to the original. The word choices will not change and the level of difficulty will remain the same. However, the revised version will include synonyms to each answer choice. In this way, the ESL student is still being exposed to the same material as a native speaker of English, is still required to critically analyze the sentence and word choices in order to decide on the best answer, however with the inclusion of synonyms the fairness of the test becomes more balanced. If the vocabulary used in the SAT is difficult for native speakers of English at the University level, let alone at the high school level, then it only makes sense that ESL learners will have an even more difficult time understanding. ESL students have not been
exposed to as much input in English vocabulary as native speakers have, which is not to say they do not understand, they just understand at a different level.
Furthermore, more time should be allotted to ESL SAT takers to compensate for the extra time spent manipulating a language other than their own as well as factors such as jet lag and culture shock. Offering the test over two days instead of one, offering more breaks during testing, and allowing more time overall to take the test. In this way speed does not become an issue like it had become in the past before the test was modified, offering more time and including less questions. Research has shown that test-takers relying solely on the "systematic" method of test taking, in which each answer choice is critically analyzed by the student were at a disadvantage due to time restraints. On the other hand, those test takers who depended on the Holistic approach were at advantage because they simply had to look at the question, which was already familiar and easily recognized what choices were wrong, eliminating them immediately, without having to actually systematically analyze try each choice. What is really being tested, this is the important question. Is it the ability to recognize and define difficult words that ESL learners may never use and that most native speakers do not even use, or is the objective to be able to understand and actually use the words in a comprehendible way.
2. Prompt Attributes
given an incomplete sentence, with one to two missing words, that they must complete using the choices provided. The objective is To select the best choice among the five possible answers. Use tests/tasks that are more global and pertain examples from all parts of the world,
Sentence completion is administered by means of multiple-choice questions. Test takers are
reflecting the great cultural, racial and ethnic diversity among the test takers. Specifically, the test should not only cover the social, cultural and historical issues related to theWestern world, but for instance. should also include those of other parts of the world, such as Korea, China, Japan, and Indonesia,
revised version, the synonyms for each choice should also be provided. In this way the student is them as ESL students but to many native English speakers as well.
still required to critically think about the answer, is still exposed to the same words that are given .
Provide words equally difficult to those provided on the original SAT exam, however, in the
in the original SAT, but not expected to understand and be able to use words not only uncommon to
instead of one and offer more breaks during testing, to acknowledge the affects of jet lag and the overall stress caused by traveling, which the majority of ELLs must endure in order to arrive at testing sites.
Allot more time to ESL learners to complete the test so that. Offer the test over two days
Response Attribute
taker will then return to the prompt, analyze sentence structure, context, try to predict the meaning or main idea of the sentence. Then the test taker will return to the five multiple choice until the correct choice stands alone.
The test taker will read the prompt and review all five multiple choice answers. The test
answers and eliminate incorrect answers, starting with the most obvious to the least obvious,
complete understanding of what is being asked, predict the most appropriate word to complete the sentence item by through an analysis of sentence structure, context, and analyzing its he will select the correct answer from among the choices given sentence structure, context and cultural background. He may eliminate any choices given. Then,
test taker will study (review) all five choices., reread the prompts in order to ensure
4. Sample Item
Original Sample Item 1: Because King Philips desire to make Spain to dominant power in sixteenth-century Europe ran counter to Queen Elizabeths insistence on autonomy for English, _____was_____. warfareavoidable reconciliationassured
ruinationimpossible diplomacysimple conflictinevitable
Because the Empire of Japan aimed to dominate East Asia and Indochina during the World War II ran counter to the autonomy of then Republic of China initially led by Sun Yat-sen, _____was_____. warfareavoidable reconciliationassured
Revised Sample Item 1:
ruinationimpossible conflictinevitable
diplomacysimple
There is no doubt that Larry is a genuine_____: he excels at telling stories that fascinate his listeners. pilferer braggart dilettante
Original Sample Item 2:
prevaricator There is no doubt that Larry is a genuine_____: he excels at telling stories that fascinate his listeners. braggart dilettante (amateur) pilferer (thief) raconteur Revised Sample Item2:
prevaricator (liar)
raconteur (noted story-teller)
Revised Sample Item 1:
Because the Government of Dutch aimed to colonize Indonesia after its Independence Day ran counter to the autonomy of the Republic of Indonesia initially led by Soekarno, _____was_____.
(A)
reconciliationassured
(B)
warfareavoidable
(C)
ruinationimpossible
(D)
conflictinevitable
(E)
diplomacysimple
I like this but do not quite understand the prompt and am afraid to correct it without knowing what you meant :)
Intermediate Oral Grammar, Version 2 Virgiawan (Virgi) Kristianto, Scoot Marta, Daniel Johnson, and Julie Kierski
Intermediate Oral Grammar Achievement Test V2.0

Fall 2012 Daniel Johnson Julie Kierski Virgiawan Kristianto Scoot Marta
Background: This test is designed for students in an intensive English program. It is intended for students at an intermediate level. In our program, students are placed into one of six level (100-600), and this test is an achievement test for those at the 300 level. Thus, if both instruction and learning were successful over the semester, the students should be able to fulfill all requirements of the test. The test requirements were built from the 300-level (See Appendix Figure 1) as written for the Intensive English Institute at the University of Illinois at Urbana-Champaign. Our intent in designing this test was to measure students' ability to produce grammar and grammatical structures orally as well as in writing. It is very common for oral grammar production to lag behind written grammar recognition and production. Therefore, after administering this test, we should have a better understanding of students' abilities in written grammar vis--vis oral grammar. This can then inform instruction if there is an unacceptably large gap between the two. In addition, administering this test (and the studying that students will do for it) will be an additional learning opportunity to attempt to encourage students to incorporate new grammar into their everyday usage.
Description of Assessment: This oral grammar test is one part of a final achievement-based assessment of how well students meet the proficiency goals for the 300-level course set, which consists of four separate courses (Listening and Speaking; Grammar; Reading; and Writing) each taught by separate instructors. The final assessment, given once at the end of the semester, consists of a series of separate tests assessing each course/skill in an attempt to gather sufficient evidence to determine the students mastery of the overall 300-level course goals. To pass on to the next level of foreign language instructions, students should demonstrate achievement of very well or satisfactorily on each test. A students end-of-term achievement scores in each of the areas will be combined together and used along with the teachers classroom assessments to determine the students placement level for the next term. The decision is a collective agreement by the teachers. Therefore, it is possible for a student to receive an assessment of inconsistently or unsatisfactorily in one test but still pass on to the next level. This oral grammar test will assess students ability to produce specific grammatical forms in the context of communicative tasks. The format of this oral test does not allow for the best assessment of certain discrete grammar forms and all of the benchmarks at the 300 level. Therefore, a separate written grammar test assessing those benchmarks will be given to the students; this test will also contribute to the final overall assessment. Since the oral grammar test items are communicative tasks, the data gathered from this test will also be used to assess the students communicative ability, specifically listening and speaking. This is a separate assessment that will also contribute to the final overall assessment.
This spec document will only cover the oral grammar test section. This spec does not provide a rubric or discuss scoring for the listening and speaking assessment.
Description of Test Takers and Test Environment: Test takers: The students who will take this achievement task will have completed the 300-level course set at the IEI. Students generally have an intermediate level of grammar knowledge and fair communicative competence. The students are presumably familiar with the types of tasks in the test as they have encountered similar tasks in classroom activities. Environment: This test will be administered in a language laboratory equipped with an audio video system. The audio video system will be used to record students responses and to enable the evaluators to assess students responses outside the room. There will be two people inside the room, one interviewer and one student. The interviewer is the person who will give students instruction and prompts as necessary.
Objectives: Through the oral grammar component of the achievement assessment evaluation the student will demonstrate how proficient they are orally in the following grammar proficiency benchmarks for the course. Ability to ask Wh- questions accurately Relate events with clarity using a variety of verb tenses in present and past, including both simple and progressive aspects Show an awareness of countability and use appropriate nouns forms, articles, and quantifiers at least some of the time Use modals accurately to express ability, necessity, and possibility in the present and future time frames, where appropriate
General Instructions: The test will be administered by one of the students teachers (from one of the courses in the 300-level course set) so the student will be comfortable conversing with and being in a room with the administrator. The interviewer should make the student feel as comfortable as possible. The student will complete a series of eight tasks that focus on different grammatical aspects. These tasks will focus on one or two major grammatical forms and consist of an interview format, text or picture prompts, or scenarios.
Throughout the oral test, the interviewer should give the student sufficient time, if the student needs it, to think through the task. The student may need some time to examine a picture or think about the scenario. If the student is struggling with the task, the interviewer may try to elicit an answer by asking guiding questions. This should not be a problem though as the students will have encountered all the tasks before, so they will be familiar with the tasks formats. Some tasks require the interviewer to explain the instructions to the student. In the sample tasks below, these instructions are bold-faced.
Student Response Attributes: Student answers should be only in English and in complete sentences. As this is a test of English, any responses or utterances in the students first language will not be assessed. Since this is a grammar test, the content and meaning of student responses is not the assessment target and therefore will not be assessed. The grammar evaluator should bear in mind that content and meaning will be evaluated in the listening and speaking assessment. Some assessment tasks require the use of a pencil and worksheet, which the student will be given. These materials are intended only for specific tasks and not for note-taking or practice.
Scoring: All of the students utterances produced within the test room setting will be open to assessment. This includes confirmation checks, clarification requests, and any asides. This will provide good data on the students natural production of English. However, the focus of the assessment is on the language produced through the tasks. Scoring will be conducted within the conditions below: d) Two classroom foreign language teachers who will do the scoring. One of the teachers will solely focus on accurate use of the grammar orally and assign a rating to the student. Even if a student misses the prompt or context, they will only look at the spoken grammar produced, whereas the other, the LS teacher, will focus more on content, fluency, accuracy, and use of conversation and listening strategies. The teachers will be familiar with the proficiency goals to which they are grading as well as the student they are assessing. e) The oral test will also be videotaped. If there is disagreement among the two teachers or if one teacher is wavering in their judgment of proficiency for a student, he or she can access the recording. Then, together the two teachers can discuss an appropriate score
f) For each grammar component being assessed in the tasks, the student will receive a score of 1) very well, 2) satisfactorily, 3) inconsistently, or 4) unsatisfactorily. The conditions and scoring scale are listed below in the chart at the end of this document. (See Appendix Figure 2)
Since this oral grammar test is one part of a students overall final assessment, this test is a lowstakes test and does not require fine-grained distinctions and exact numerical scoring. Scoring is done through the expert judgments of experienced teachers who have been teaching the student throughout the semester. If for some reason it is necessary to find out the exact number of errors or the specific kinds of errors a student made on the test, evaluators may consult the recording. The scoring rubric also allows for flexibility in how loosely or tightly an evaluator may define the four scores from semester to semester. The student will receive a score for each of the eight grammar tasks. We suggest that scores of satisfactorily on at least six of the eight tasks defines achievement on the test. However, as flexibility is allowed, the teachers may decide that scoring satisfactorily on seven of the eight tasks defines achievement on the test; or they may decide that scoring satisfactorily on four of the eight tasks and inconsistently on two other tasks is sufficient.
SAMPLE TASKS
Task 1: Simple Present

Interviewer General Instructions: This warm-up interview should feel more like a friendly conversation than a formal interview and will be used to assess the students ability to use simple present tense. The interviewer will ask the student a few questions to engage the student in conversation. These questions will elicit students answers that use the simple present tense. The interviewer may respond to indicate that he or she is listening to the student and may ask for clarification if an utterance or meaning is unclear. However, the interviewer should not dominate the conversation. If a student is more reticent, the interviewer may have to ask more questions to elicit responses. Below are sample questions that the interviewer may ask. Easier questions should be asked at first; if the interviewer determines that the student is more advanced, then progressively more difficult questions can be asked. What do you do in your free time? Whats your favorite restaurant and why? What is your home country like? What do you like about living in America? Task Instructions to Give Students: There are no instructions for the students. This warm-up interview should be a natural conversation between the interviewer and the student. Sample Student Responses: I like to watch movies. Saudi Arabia is very hot. I eat pizza everyday. How to Formulate New Tasks: New questions to ask the student can be formulated based on the content covered throughout the courses. Questions should address topics that the student is familiar with and has gained lexical knowledge of through lessons taught. Questions should elicit answers using the present simple tense.
Task 2: WH questions
Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of WH questions. This test should last no longer than five minutes. In this task, the evaluator will 8) pose a question to the student about a certain character that he or she is very familiar with, then take that role as the students chosen character, 9) direct the student to ask WH questions with regards to his or her chosen character, 10) ask the student to create one question for practice. This practice question will not be scored, 11) answer the WH questions of the student using complete sentences instead of chunk of words, 12) prompt the student for what he or she should ask, using one word only such as hobby or address if the student consistently struggles to create questions, 13) record the response of the student, 14) end the task if the student stops producing the intended responses or the time is over. Task Instructions to Give Students: In this part of the task, imagine a character that you are familiar with and curious about such as your favorite celebrity. Ask WH questions about that character to the interviewer. The WH questions must use simple present tense. Sample Student Responses: Where do you live? What are your hobbies? How can I become famous like you? How to Formulate New Tasks: When choosing another situation, make sure the situation is widely known for triggering the students participation in asking questions using simple present tense. The chosen situation needs to be really familiar for both students and the evaluator and to focus on the use of the grammar rather than other aspects of language. Avoidance needs to be made upon situations which require too much technical vocabulary.
Task 3: Simple Past

Interviewer General Instructions: This portion of the test will focus on how well the student can correctly ask questions in the simple past. This task should last no longer than five minutes. In this task, the interviewer will hand the mock schedule (Students Sheet) to the student, then read the directions to the student. The student should be allowed 15-20 seconds to look at the schedule and think about what they want to say and then respond orally. This is an information gap activity, wherein the students job is to complete the blank portions of the schedule by asking questions. The interviewer will respond to the questions using the Interviewers Sheet. Ideally, the student will employ a variety of verbs and question words. Be as clear as possible that this what John DID on Monday and Tuesday. Task Instructions to Give Students: Look at the schedules. They show what John, a university student, did on Monday and Tuesday. There are several blanks in the schedule. Ask questions so that you can fill in all the blanks. Remember, the schedules show what John DID on Monday and Tuesday. Sample Student Responses: What time did Johns chemistry class start on Monday? What did John do from 11:00 AM to 1:00 PM on Tuesday? When did Johns soccer practice end on Monday? What class did John have at 10:00 AM on Tuesday?
STUDENTS SHEET
Johns Schedule
9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-_:__ PM Monday Spanish Class Tuesday 9:00-10:00 AM 1:00-2:00 PM 2:00-_:__ PM _:__-8:30 PM Spanish Class Lunch Study
Geography Class Chemistry Class Soccer Practice Study
11:00 AM 1:00 PM
10:00-11:00 AM
__________ Class Math Class Dinner
11:30 PM-12:30 AM
_:__-11:30 PM
8:30 PM 12:00 AM 12:00-1:30 AM
INTERVIEWERS SHEET
Monday 9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-5:00 PM _:__-11:30 PM 11:00 AM 12:00 PM Spanish Class Lunch Tuesday 9:00-10:00 AM Spanish Class English Class Lunch Nap
Geography Class Chemistry Class Soccer Practice Dinner Study
11:00 AM 1:00 PM 1:00-2:00 PM 2:00-3:30 PM 8:30 PM 12:00 AM 12:00-1:30 AM 7:30-8:30 PM
10:00-11:00 AM
Math Class Dinner Study
11:30 PM-12:30 AM
Watch TV
Watch Movie
How to Formulate New Tasks: If creating a new schedule or activity, ensure that the fictitious persons name will be easy to pronounce and clearly identifiable as either female or male to avoid difficulties in pronunciation or subject pronoun choice. Choice of classes and activities in the schedule should be tailored such that they will be recognizable to students in order to retain the focus on grammar and not on vocabulary. If the day of the test is known in advance, it might be beneficial to make the schedules for the two days preceding the test. If an activity other than a schedule is chosen, use one which the student will be familiar with and will elicit various types of Wh- questions and verbs.
Task 4: Modals (Possibility)

Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of modal of degree of certainty in present tense. This test should last no longer than five minutes. In this task, the evaluator will 6) ask the student to imagine a certain situation in which he or she can make guesses such as guessing gifts, the possibility of a certain team winning a game, or weather prediction, 7) have the student create 5 guesses to which you must not respond, 8) forbid the students to jot down his or her guesses before he or she produces them orally, 9) prompt the student if he or she pauses for certain amount of time, 10) end the task if he or she struggles to construct the guess or the time is over. Task Instructions to Give Students: In this part of the task, construct 5 sentences using modals of degree of certainty in present tense. In this part of the task, you will 3) imagine your birthday party 4) guess orally 5 gifts that you get using modals of degree of certainty in simple present tense Sample Student Responses: I may get a book. My parents might buy me an iPod. My boyfriend could give me a rose. How to Formulate New Tasks: Other scenario might be applicable for this task. However, it is necessary to set a scenario in which people usually make guesses such as guessing gifts, a weather report, a football teams possibility to win a game, etc. Have the student choose one of the scenarios that he or she really understands.

Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use the present progressive and control it in conversation. This task should last no longer than five minutes. In this task, the evaluator will hand a picture to the student, then read the description. While listening closely to the description, the student should look at the picture. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. This is an adaptive test. The test will begin with the student speaking freely, but if a student consistently struggles to answer, the interviewer can help direct the students focus by pointing to a picture and ask a WH- question. For example, What is this lady doing? What is the shoe store owner doing? Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture, the interviewer can end the test. Task Instructions to Give Students: Look at the picture below. Imagine that you are in your apartment. You are looking out your window and this is what you see. Describe all the things that you see that are happening. You have 15 seconds to review the picture before you speak.
(Communicative Grammar Practice, Leo Jones, 1985)
Sample Student Responses: A boy is selling newspapers. A lady is walking her dog. The man with a moustache is sweeping. How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card. This would create a more challenging task for students though since the students wont have the 15 seconds to compose their students as in the picture task. Additionally, the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. When choosing another picture, its important to make sure that the picture has enough images to produce a wide variety of sentences in the progressive form. Whatever image is used, its important that the students have general knowledge of the vocabulary in order to retain the focus on grammar, otherwise it could become a test of vocabulary knowledge.
Task 6: Past Progressive

Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use past progressive and control it in conversation. This task should last no longer than five minutes. In this task, the interviewer will hand the picture over to the student then read the description. While listening closely to the description, the student should look at the picture. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. The test will begin with the student speaking freely, but if a student consistently struggles to answer, the interviewer can help direct the students focus by pointing to a picture and ask a W-H question. For example, What was this man doing? Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture, the interviewer can end the task. Task Instructions to Give Students: Look at the picture below. Mr. and Mrs. Gold invited some friends to their house for the weekend. A thief stole Mrs. Golds jewelry at midnight. Describe all the things that the guests were doing when the thief stole the jewelry at midnight. You have 15 seconds to review the picture before you speak.
(Azar Beginner Grammar Textbook Series)
Sample Student Responses: When Ms. Orange was reading, a mouse was eating cheese. (When the thief stole the jewelry), Mr. Blue was watching TV. Mr. and Mrs. Gold were sleeping. How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card. This would create a more challenging task for students though since the students wont have the 15 seconds to compose their students as in the picture task. Additionally, the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. When choosing another picture, its important to make sure that
the picture has enough images to produce a wide variety of sentences in the progressive form. Whatever image is used, its important that the students have general knowledge of the vocabulary in order to retain the focus on grammar, otherwise it could become a test of vocabulary knowledge. If names are assigned to people in the picture, as in this sample task, make sure the names of the fictitious characters can be pronounced easily like this task does with colors. This task can also be made into a 1-on-1 conversation by asking the student questions about what they did yesterday during certain times. For example, the teacher could imagine that they are a policeman and interview the suspect (the student) by asking a variety of questions that should product the progressive aspect such as What were you doing at 8 AM yesterday? or We found evidence of food in your car. What were you eating in your car?
Task 7: Modals (Necessity)

Interviewer General Instructions: This portion of the test will focus on the students knowledge and control of necessity modals. This test should last no longer than five minutes. In this task, the evaluator will read the prompt to the student. The student will have 15-20 seconds to think before beginning to respond orally. If the student seems to be having serious difficulties with inventing rules (i.e. generating content), some prompting may be appropriate, such as What about homework? or What about talking in class? Care should be taken not to suggest too many forms to the student, however. Task Instructions to Give Student: Imagine you are a teacher in a high school. It is the first day of class. Think of some rules that you will tell your class on the first day. Sample Student Responses: You must not smoke in class. You must turn your cell phone off. You have to bring a textbook and a pen to class. How to Formulate New Tasks: Other scenarios are, of course, imaginable for this language feature. Whatever might be chosen, ensure that it is possible to elicit various necessity modals, bearing in mind that avoidance, whether intentional or unintentional, may intrude. Try to structure the task in such a way that it is not possible (or not easy) to use ability modals like can and cant. If students are not comfortable with necessity modals, they may try to avoid them altogether by using ability or advice modals. Both activity and directions can be useful tools to attempt to prevent this sort of avoidance.
Task 8: Articles / Quantifiers

Interviewer General Instructions: In this task, the student will be given a photo of an open refrigerator/freezer with various kinds of food inside of it. The student will describe the items in the photo. The student will be assessed on the correct use of articles and quantifiers with the accompanying count and non-count noun(s). This task will also produce data for the assessment of verb agreement in the student's utterances. The student may be creative in their descriptions and produce longer sentences, but only the articles, quantifiers, and verb agreement will be assessed for the oral grammar assessment. This task should take no longer than five minutes. The student should describe at least 6 items in the photo. If the student only produces statements using count nouns, the interviewer should prompt the student to describe a non-count item, and vice versa. If the student only describes plural nouns, then the interviewer should prompt the student to describe a single item (to elicit the use of the articles "a" or "an"), and vice versa. If the student is having trouble producing the sentences, the interviewer may elicit responses by using guiding statements or questions such as the ones below: Interviewer (pointing to an item): Can you describe this? Or if the student needs more guidance, Interviewer: Are there any apples in the fridge? Student: Yes, there are. Interviewer Follow-up: How many apples are there? Student: There are a lot of apples in the fridge. Task Instructions to Give Students: Describe what you see in this picture. You may use words such as "a lot of", "lots of", "many", "some", "a", and "an".
Sample Student Responses: There is fruit in the bottom drawer. There are three peppers in the fridge. The cake is on the top shelves. How to Formulate New Tasks: Other picture scenarios, such as a shelf in a store or a room in a house, may be used for this task. Whatever picture that is used should show a variety of count and non-count nouns; these nouns
should be ones that the students should have in their lexical knowledge through lessons that were taught in the 300 level or in the previous 100 or 200 level courses. Pictures can be black and white line drawings, color drawings, or color photos as long as the picture is clear and large enough to see the individual items in the picture.
Appendix
Figure 1
Figure 2
SCORE Grammar
A few errors, overall communication of ideas is clear
Very Well
Satisfactory
Several errors, overall communication of ideas is still clear
Inconsistently
Significant and consistent errors that sometimes confuse meaning
Simple Present
Task 1:
Many errors that making meaning unclear. Listeners attention is diverted to the errors from the message.
Unsatisfactorily
WH- Questions
Task 2: Task 3:
Simple Past Possibility Modals
Task 4:
Present Progressive
Task 5:
Task 6: Past Progressive Task 7: Necessity Modals Task 8: Articles
Quantifiers
Understanding Instructions, Version 2 Katie Honken, Sophie Qin, Becky Menendez, and Mica Swyers
Introduction
This test is designed to serve as a supplement to the English Proficiency Test (EPT) given at the University of Illinois Urbana-Champaign for incoming non-native speakers of English. Assuming incoming students have a sufficient score to enter the university, this test targets intermediate-level English-language learners who would be expected to succeed in American university-level courses. While the EPT tests writing skills and pronunciation, this supplement will assess students ability to succeed in a university course and classroom setting by following instructions in a variety of formats. These test items are representative of a variety of items that would appear of the test to reflect different teachers preferences for formatting and instructions, including accent, speed of delivery, written vs. oral, etc. Students who do not demonstrate sufficient ability will be placed in a basic classroom survival skills class in conjunction with their ESL courses at the University of Illinois. This EPT supplementary section is necessary to reduce the influence of students ability to follow instructions on their performance in a university-level course (i.e., losing points due to inability to follow instructions properly).
General Objectives
Specific Objectives
5. 6. 7. 8. Assessing students ability to understand written directions of assignment. Assessing students ability to understand and follow oral directions for in-class work. Assessing students ability to understand and apply a rubric. Assessing students ability to understand written feedback.

Item 1: Formatting
Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in a written format: detailed instructions of editing a paragraph will be provided to examinees, and those instructions are required to be understood by students in university-level courses. The examinees will face four types of formats in screenshots, among them only one format is correct and the other three are with different problems including spacing, margin, size or editing mistakes in the title. The key screenshot correctly follows the instructions provided in the item stem. The three distractors will depict various formats that arise from misunderstanding of the instructions. Response Attributes (RA) To select the right format from the four options, the examinees need to read through the instructions. Then they will the information from instructions to the formats displayed in screenshots, and select the format that meets all requirements. The computer testing software will record the result and categorize it as correct or incorrect. Correct answer earns 2 points while incorrect answers earn 0 points. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. C: The title is not bold and the text size is 10pt. D: The font is not Times New Roman. Sample Question 1 (Written): In a writing class students are required to edit the provided passage based on the following format: Font: Times New Roman (This is what Times New Roman looks like.) Spacing: Double-spaced Size: 12pt (This is what 12pt. font looks like.) Title: Times New Roman, bold, centered Which of the following formats meets the teachers requirement? ________
Choice A
Choice B
Choice C
Choice D
Prompt Attributes (PA)
This is a four-option multiple choice test item which tests students ability to hear and understand oral formatting instructions, such as those given in class. The item stem will be given in an oral format: detailed instructions of editing a paragraph will be provided to examinees, and those instructions are required to be understood by students in university-level courses. Students will be allowed to listen to the clip twice. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits.) The examinees will face four types of formats in screenshots, among them only one format is correct and the other three are with different problems including spacing, margin, size or editing mistakes in the title. The key screenshot correctly follows the instructions provided in the item stem. The three distractors will depict various formats that arise from misunderstanding of the instructions.
Response Attributes (RA) To select the right format from the four options, the examinees need to read through the instructions. Then they will the information from instructions to the formats displayed in screenshots, and select the format that meets all requirements. The computer testing software will record the result and categorize it as correct or incorrect. Correct answer earns 2 points while incorrect answers earn 0 points. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. C: The title is not bold and the text size is 10pt. D: The font is not Times New Roman.
Sample Question 2 (Oral):

The examinee(s) should hear: Your paper should be double-spaced in Times New Roman, 12 point font. The title should be Times New Roman as well, bolded and centered at the top of the page.
Choice A
Choice B
Choice C
Choice D
Answer Key:
Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in oral format: the examinee will listen to a sound file of a teacher giving instructions for an in-class assignment. These will be instructions that a student would hear in a university-level course, for tasks such as groupwork, peer review, or an in-class assignment. Test-takers will be allowed to replay the clip once. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits.) The examinee faces a computer monitor divided into quadrants (one for each choice), each with a different illustration. The key picture correctly illustrates the directions given in the oral item stem. The three distractors will depict different scenarios that could arise from following the instructions incorrectly or incompletely. The drawings below are mock-up pictures that would not actually be included in the actual test. Instead, students might see pictures of real humans depicting different scenarios resulting from following the instructions correctly/incorrectly. They could also be given short video clips of groupwork scenarios. In either case, test-takers still have to determine which choice best illustrates the correct following of instructions. Sound clips should feature different voices (male vs. female, different accents, etc.) and different speeds of delivery. This will represent students receiving instructions from different teachers in a university-setting. Response Attributes (RA) To select the correct choice of the four options, the examinee will have to listen to the sound file. Then s/he will have to process the information and select the picture (by clicking) that best illustrates the instructions that were given. The computer testing software will record the item that the student selected and categorize it as correct or incorrect. Sample Item The examinee(s) should hear: Now, get into groups of three or four and come up with a paragraph responding to the question on the board. You should only turn in one paragraph per group at the end of the class.

0 points
Support
Grammar

Length

Option B:
Option C:
Option D:
Understanding Instructions, Version 2.1 Katie Honken, Sophie Qin, Becky Menendez, and Mica Swyers
Introduction
This test is designed to serve as a supplement to the English Proficiency Test (EPT) given at the University of Illinois Urbana-Champaign for incoming non-native speakers of English. Assuming incoming students have a sufficient score to enter the university, this test targets intermediate-level English-language learners who would be expected to succeed in American university-level courses. While the EPT tests writing skills and pronunciation, this supplement will assess students ability to succeed in a university course and classroom setting by following instructions in a variety of formats. These test items are representative of a variety of items that would appear of the test to reflect different teachers preferences for formatting and instructions, including accent, speed of delivery, written vs. oral, etc. Students who do not demonstrate sufficient ability will be placed in a basic classroom survival skills class in conjunction with their ESL courses at the University of Illinois. This EPT supplementary section is necessary to reduce the influence of students ability to follow instructions on their performance in a university-level course (i.e., losing points due to inability to follow instructions properly).
General Objectives
Specific Objectives
9. Assessing students ability to understand written directions of assignment. 10. Assessing students ability to understand and follow oral directions for in-class work. 11. Assessing students ability to understand and apply a rubric. 12. Assessing students ability to understand written feedback.

Item 1: Formatting
Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in a written format: detailed instructions of editing a paragraph will be provided to examinees, and those instructions are required to be understood by students in university-level courses. The examinees will face four types of formats in screenshots, among them only one format is correct and the other three are with different problems including spacing, margin, size or editing mistakes in the title. The key screenshot correctly follows the instructions provided in the item stem. The three distractors will depict various formats that arise from misunderstanding of the instructions. Response Attributes (RA) To select the right format from the four options, the examinees need to read through the instructions. Then they will the information from instructions to the formats displayed in screenshots, and select the format that meets all requirements. The computer testing software will record the result and categorize it as correct or incorrect. Correct answer earns 2 points while incorrect answers earn 0 points. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. C: The title is not bold and the text size is 10pt. D: The font is not Times New Roman. Sample Question 1 (Written): In a writing class students are required to edit the provided passage based on the following format: Font: Times New Roman (This is what Times New Roman looks like.) Spacing: Double-spaced Size: 12pt (This is what 12pt. font looks like.) Title: Times New Roman, bold, centered Which of the following formats meets the teachers requirement? ________
Choice A
Choice B
Choice C
Choice D
Prompt Attributes (PA)
This is a four-option multiple choice test item which tests students ability to hear and understand oral formatting instructions, such as those given in class. The item stem will be given in an oral format: detailed instructions of editing a paragraph will be provided to examinees, and those instructions are required to be understood by students in university-level courses. Students will be allowed to listen to the clip twice. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits.) The examinees will face four types of formats in screenshots, among them only one format is correct and the other three are with different problems including spacing, margin, size or editing mistakes in the title. The key screenshot correctly follows the instructions provided in the item stem. The three distractors will depict various formats that arise from misunderstanding of the instructions.
Response Attributes (RA) To select the right format from the four options, the examinees need to read through the instructions. Then they will the information from instructions to the formats displayed in screenshots, and select the format that meets all requirements. The computer testing software will record the result and categorize it as correct or incorrect. Correct answer earns 2 points while incorrect answers earn 0 points. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. C: The title is not bold and the text size is 10pt. D: The font is not Times New Roman.
Sample Question 2 (Oral):

The examinee(s) should hear: Your paper should be double-spaced in Times New Roman, 12 point font. The title should be Times New Roman as well, bolded and centered at the top of the page.
Choice A
Choice B
Choice C
Choice D
Answer Key:
Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in oral format: the examinee will listen to a sound file of a teacher giving instructions for an in-class assignment. These will be instructions that a student would hear in a university-level course, for tasks such as groupwork, peer review, or an in-class assignment. Test-takers will be allowed to replay the clip once. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits.) The examinee faces a computer monitor divided into quadrants (one for each choice), each with a different illustration. The key picture correctly illustrates the directions given in the oral item stem. The three distractors will depict different scenarios that could arise from following the instructions incorrectly or incompletely. The drawings below are mock-up pictures that would not actually be included in the actual test. Instead, students might see pictures of real humans depicting different scenarios resulting from following the instructions correctly/incorrectly. They could also be given short video clips of groupwork scenarios. In either case, test-takers still have to determine which choice best illustrates the correct following of instructions. All photos and video clips will be in color and will be captured in the same classroom. The background setting of the pictures/videos will be simple and uncluttered. All background elements not essential to the question item will be identical in all four choices. Each picture/video will draw from the same group of student actors (although the total number of actors may vary from scene to scene, depending on the prompt). However, if the question item specifically mentions, for example, a question written on the board, this will appear in the pictures/video clips. Sound and video clips will be long enough to give test takers a sense of how the directions are being followed, about 15-20 seconds each. Group dialogues will be based on actual in-class activities from a variety of university classes. Sound and video clips should feature different voices (male vs. female, different accents, etc.) and different speeds of delivery. This will represent students receiving instructions from different teachers in a university setting. Response Attributes (RA) To select the correct choice of the four options, the examinee will have to listen to the sound file. Then s/he will have to process the information and select the picture (by clicking) that best illustrates the instructions that were given. The computer testing software will record the item that the student selected and categorize it as correct or incorrect. Sample Item The examinee(s) should hear: Now, get into groups of three or four and come up with a paragraph responding to the question on the board. You should only turn in one paragraph per group at the end of the class.

0 points
Support
Grammar

Length

Option B:
Option C:
Option D:
The Barry Spec -- Testing Pragmatics, Version 2 Alberto Domnguez, Widya Ratna Kusumaningrum, and Yelena Forrester Knezevic
Pragmatic Test Specs: The Barry Item
EIL 460, Fall 2012 Alberto Domnguez Yelena Forrester Knezevic Widya Ratna Kusumaningrum
Language Skills: Speaking Pragmatics Item types: - Making small talk - Excusing oneself
- Requesting information - Expressing concern Methods: - Oral interview - Conversation - Short answer
Background:
This test will be administered in a pragmatics course for advanced-level ESL students. The objectives for the course will be to increase students sociolinguistic, strategic and discourse competence; this test is designed to measure students proficiency in these areas. Students should be familiar with the types of tasks that will be tested before taking this test. However, because this is not a diagnostic or placement test, no previous knowledge of these topics is expected before students begin the class. Instructors will introduce the concepts during the course. Testing will occur after the instructor feels that students have reached the appropriate level. Participants and Setting: This test is designed for advanced-level ESL students. They will be tested on topics that have been introduced by their teacher during the course of the semester. The test will be administered in a one-on-one setting, with one examiner and one test-taker at a time. The examiner will not be the students current instructor, but a teacher of another section of the course.
General specs:
These specs are applicable in a classroom setting where students are aware of the pragmatic content and have agreed to be tested on potentially sensitive subjects. The testtaker will have signed off on something like the following:
This class will include tasks that test my ability to conduct small talk, request and give information, excuse myself from a situation, and express concern to friends about their potentially harmful behavior. I understand that some of these topics may be sensitive. I understand that I will not be tested on whether my answers follow strict patterns or formulas, nor on whether I agree with the implications of certain test items. Rather, I will be evaluated based on the politeness and appropriateness of my responses.
General Objectives:
Test takers will use their rationale to demonstrate integrated competences (strategic, discourse, and sociolinguistic). Their responses will be assessed based on adequacy and appropriateness of the language context. Each examiner will evaluate test-takers responses based on a scoring rubric. General Scoring Rubric: The speaking tasks will be assessed by the examiner using the rubric provided in this section. The test item score will consist of the summation of the subscores for the three categories in the rubric; strategic, discourse and sociolinguistic competence. The test items will have a maximum possible score of 12 points, with a minimum of 0 (if the test-taker does not attempt any answer). Competence Type Strategic Fulfills all of the Fulfills most of objectives of the objectives the task. of the task; or addresses all of the objectives, but does not fulfill all of them. Clear, effective and logical organization of response. Fulfills all or most of the following criteria: Correctly judges the formality of a situation, and uses the Somewhat clear, effective, and logical organization and response. 4 3 Fulfills some of the objectives of the task; or addresses most of the objectives, but fulfills only a few of them. Few examples of clear, effective and logical organization. 2 Does not fulfill any of the objectives of the task; or addresses a few objectives, but does not fulfill any of them. Minimal, examples of clear, effective and logical organization. Fulfills all or most of the following criteria: Incorrectly judges the formality of a situation, and uses the 1
Discourse Sociolinguistic
Fulfills all or most of the following criteria: Judges the formality of a situation with minor errors, and uses most of the
Fulfills all or most of the following criteria: Judges the formality of a situation with major errors, and uses only a few of the
* A score of 0 can be given if the test-taker does not attempt any answer. A is 100-90
appropriate forms. Uses greetings and farewells as appropriate. Demonstrates advanced understanding of the relationship (as defined in the test item) between testtaker and interlocutor.
appropriate forms. Mainly uses greetings and farewells as appropriate. Demonstrates adequate understanding of the relationship (as defined in the test item) between testtaker and interlocutor.
appropriate forms. Exhibits major errors in greetings and farewells. Demonstrates little understanding of the relationship (as defined in the test item) between testtaker and interlocutor.
inappropriate forms. May also be rude. Uses greetings and farewells inappropriately . Demonstrates minimal understanding of the relationship (as defined in the test item) between testtaker and interlocutor.
** Percentages will be calculated from the aggregate score of the student on all test items and score will be reported using the customary letter grade for classroom assessments. B is 89.9-80 C is 79.9-70
D is 69.9-60
F is 59.9 or below Time Allotment: 30 minutes
Interviewer General Instructions:
This section should be read by the examiner to test-takers before they participate in this test: You have been studying pragmatics in this course. The subjects of the test items may make you uncomfortable, but it is important to remember that the focus of this test is
pragmatics. You will be tested on your ability to appropriately interact with other people in English. You will not be graded on whether you agree with the items or not. The test items should be answered only in English, and anything in another language will not be graded. Remember that the goal of the test is to assess how well you follow the sociolinguistic, strategic, and discourse norms that we have discussed in class.
Small talk
Creating new items: writers can create new sample starting lines. To increase the difficulty of the item, the examiner should be directed not to prompt the test-taker when the test-taker does not issue a satisfactory response. To reduce the item difficulty, the examiner should prompt the test-taker whenever necessary, and should avoid introducing material such as My son has been sick (since such material is likely to distract the reader emotionally).
1. Prompt Attribute: The examiner and test-taker will roleplay a situation in which small talk with strangers could be likely to occur. The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before ending the conversation. Then the examiner will initiate the conversation. (This task could come at the beginning of an oral interview with the examiner.)
1. Response Attribute: Each speakers responses should proceed logically from the previous utterance. The test-taker will respond politely and make a few generic comments of their own. The conversation should not stray outside the boundaries of typical small talk (sports, weather, some mutual obstacle, etc.). If a test-taker is not well-versed in a topic (e.g. sports), they should be able to go along anyway, either assenting and asking questions or admitting lack of familiarity. (For example, the test-taker could say in response to a question about a sports team, I havent really been following them. How was their last game?) One-word utterances (Yeah, Really?, Uh-huh?, etc.) are appropriate as long as they offer a conversational opening for the interlocutor. The test-taker should not ask questions about the examiners appearance or personal life. When it comes time to end the conversation, the test-taker will do so gracefully, with politeness but without overfamiliarity. 1. Sample starting lines:
You are standing on a street corner when the person next to you, a stranger, starts up a conversation. Respond using small talk strategies. - Wow, this rain is really something. - How bout them Steelers? (substitute a local sports team) - Man, I cant wait to get out of here (or: get home). - Can you believe this? The line hasnt moved in ten minutes.
- That house already has Valentines Day decorations up. It gets earlier and earlier every year. (You will have 2 minutes to conduct this short conversation.)
1. Sample Response Score of 12: Examiner: Wow, this rain is really something.
Test-taker: Yeah, you got it. Glad I remembered my umbrella.
Examiner: Thats too bad. I have to pick up my daughter from school. Test-taker: I hope so too. Well, stay dry! Examiner: You too. Score of 9:
Examiner: Do you know if its supposed to keep raining all day?
Test-taker: No, I didnt check the weather. But it looks like itll be like this for a while.
Examiner: Wow, this rain is really something. Test-taker: Yeah, it really is. I dont like it. Test-taker: Um, no, sorry.
Examiner: Do you know if its supposed to keep raining all day?
Examiner: I hope it stops soon. I have to pick up my daughter from school.

Examiner: Thanks, you too. Score of 6: Examiner: Wow, this rain is really something. Test-taker: Yeah, I guess. Test-taker: Maybe. Examiner: Bye. Score of 3:
Test-taker: Good luck.
Examiner: Do you know if its supposed to keep raining all day? Test-taker: Okay, bye.
Examiner: I hope it stops soon. I have to pick up my daughter from school.
Examiner: Wow, this rain is really something. Test-taker: No, its not bad at all. Test-taker: Probably. Examiner: Okay.
Examiner: Do you know if its supposed to keep raining all day? Test-taker: Excuse me.
Examiner: Thats too bad. I have to pick up my daughter from school.
2. Prompt Attribute: The examiner and test-taker will roleplay a situation in which the test-taker is responding to small talk initiated by a friend. The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before ending the conversation. Then the examiner will initiate the conversation. (This task could come at the beginning of an oral interview with the examiner.)
2. Response Attribute: Each speakers responses should proceed logically from the previous utterance. The test-taker will respond politely and familiarly, inquiring as to the friends own opinions. The conversation should not stray outside the boundaries of typical friendly small talk (sports, weather, friends or family, some mutual obstacle, etc.). If a testtaker is not well-versed in a topic (e.g. sports), they should be able to go along anyway, either assenting and asking questions or admitting lack of familiarity. (For example, in response to a question about sports, the test-taker could say, Well, you know Im not much of a sports person. How was the last game?) One-word utterances (Uh-huh, Yeah, etc.) are appropriate as long as they offer a conversational opening for the interlocutor. When it comes time to end the conversation, the test-taker will do so familiarly, using a Good seeing you or See you around type of closing statement. 2. Sample starting lines: Your friend has just started up a conversation with you. Respond using small-talk strategies. - Hey, man, hows it going? - Hey, I havent seen you around much lately. How have you been? - Can you believe this? She said she was going to give back our grades in a week, and its been a week and a half. - Hi! I didnt see your brothers at the game last week. How are they doing? (You will have 2 minutes to conduct this short conversation.)
3. Sample Response Score of 12: Examiner: Hey, I havent seen you around much lately. How have you been? Test-taker: Oh, you know, keeping busy. Things have been crazy at work. Examiner: Thats too bad. Hows your family? Test-taker: Theyre doing all right. How about yours?
Examiner: Theyre okay, but my son has been sick. Test-taker: Oh, Im sorry to hear that. Examiner: Will do. Examiner: Yeah, hes feeling better now. Test-taker: See you around. Examiner: See you.. Score of 9: Test-taker: Thats good. Give him my best.
Examiner: Fine. Hows school going? Test-taker: Its busy, but all right.
Examiner: Hey, I havent seen you around much lately. How have you been? Test-taker: Fine, how about you?
Test-taker: Theyre doing okay. How about yours? Test-taker: Oh, sorry. Examiner: Yeah. Examiner: Yeah, hes feeling better now.. Test-taker: Thats good. Examiner: See you.. Score of 6: Test-taker: See you around.
Examiner: Theyre okay, but my son has been sick.
Examiner: Thats good. Hows your family?
Examiner: Hey, I havent seen you around much lately. How have you been?
Examiner: Hows school going? Test-taker: Theyre fine. Test-taker: Oh? Test-taker: Its fine.
Test-taker: Ive been fine.
Examiner: Thats good. Hows your family?
Examiner: Yeah, hes feeling better now.. Test-taker: Okay, good. Examiner: See you. Score of 3: Test-taker: I bet. Ill see you later.
Examiner: Glad to hear it. Lately my son has been sick.
Examiner: Yeah. We were pretty worried.
Examiner: Lately my son has been sick. Test-taker: Oh. Test-taker: ...
Examiner: Hows your family?
Examiner: Hows school going? Test-taker: Um, I dont know.
Examiner: Hey, I havent seen you around much lately. How have you been? Test-taker: Okay.
Test-taker: How is your family?
Examiner: We were pretty worried.
Examiner: Yeah, hes feeling better now..
Test-taker: Bye. Examiner: Bye.
Requesting Information
1. Prompt Attribute: This item assesses ability to request information. The prompt will consist and interlocutor. Then there will be another sentence which will establish the need the test-taker has and how the interlocutor is involved in the requesting of the information. The relationship
of a paragraph. The paragraph will indicate the relationship, or lack thereof, between testtaker
will inform how the information should be requested: level of politeness, proper greeting, etc. The test-taker will then be prompted to orally request the information needed for the situation described in the prompt. Test items built from this description may consist of diverse topics in order to accurately access the students ability to request information in many situations.
1. Response Attribute: The item will be responded to orally, simulating how the test-taker would request information. The response will include a greeting. After this, there will be a question worded appropriately based on the interlocutor being addressed. The response should avoid rudeness while being concise, appropriate and effective in conveying meaning. 1. Sample Item (You have 3 minutes to read and think over the item and 1 minute to respond): You have gotten off at the wrong bus stop and are late to a dinner party at your friends home. You have never been in the neighborhood where you are currently lost. As you enter a gas station you see a middle-aged woman at the counter and approach her. Ask the woman for directions to your friends address in an appropriate and polite way.
1. Sample Responses:
Score of 12: Excuse me, sorry to bother you but I was on the bus and I seem to be lost. I would appreciate it if you could help me. I am trying to find 123 Fake Street, can you please tell me how to get there?
Score of 9: Hello, can you please help me find 123 Fake Street? Im not sure I can walk from here, could I also take the bus? Score of 6: I need your help. Where is 123 Fake Street? Im a little late. Score of 3: How do I get to 123 Fake Street?
2. Prompt Attribute: This item will test the test-takers ability to give information to others. The prompt will consist of an interaction between the test-taker and another person. Alternately, the prompt may be presented as a dialogue leading up to the target interaction. The interlocutor in the item will vary in age and familiarity with the test-taker, and the variation will be stated in the prompt. The setting of the interaction will influence the formality of the prompt conversation. A topic of interest will be brought up in the prompt. The interlocutor will then request information from the test-taker relating to this interest. The request will come in the form of a question. The knowledge of the test-taker will vary, the prompt may tell the test-taker that she or he does not know the answer and that they must respond to the question by stating this in a manner appropriate for continuing the conversation. 2. Response Attribute: The item will be responded to orally. The test-taker may respond by stating a lack of knowledge of the information in an appropriate manner if indicated in the prompt. If this is not stated in the prompt the test-taker will provide the information requested of him or her politely and concisely. The response must logically address the interlocutor and fully answer the question. A full answer will consist of a response which provides the interlocutor with information they can use to address their concern.
2. Sample Item (You have 3 minutes to read and think over the question and 2 minutes to respond): You are drinking coffee at your friends apartment. Her friend, whom you just met, overhears you mention an upcoming concert at a local bar. He asks you whether the tickets are already for sale and if not when they will be on sale. You do not know if the tickets are being sold or when they will begin to be sold. Convey this to him and propose some other way he may find the information. 2. Sample Responses: Score of 12: You know Im not sure when the tickets will go on sale. I do know that the bar has flyers around campus and maybe you could call them and ask them about ticket sales. Sorry I couldnt be more helpful.
Score of 9: I wish I knew. Im sure you could contact the bar and find out. I hope you can get tickets in time. Score of 6: Sorry, I dont know when they go on sale. I wasnt planning on attending the concert. Score of 3: Oh yeah, I dont know about anything about that.
Excusing Expression
1. Prompt Attribute The test item measures the students ability to excuse themselves. With diverse topics, the test item will assess test takers sociolinguistics, strategic and discourse competences. The test item asks them to respond using a proper greeting & closing as well as proper usage of politeness and familiarity levels. The prompt involves an interpersonal relationship (schoolmate, neighbourhood, etc). The prompt begins with the examiner reading a sample paragraph about assigned topics.
1. Response Attribute The test taker will respond to the item orally. Their response will be graded based on the use of a proper greeting-closing and the appropriateness of the politeness and familiarity level. The Test-taker also needs to incorporate the interpersonal relationship factor into the response which will differ from one response to another. The response should be in the form of a short answer. Sample Item 1 (You have 2 minutes to read and think and 1 minute to respond) Imagine that you are living in a suburb and have a good relationship with your neighbors. You usually gossip with them about everything. Last night, you heard a police siren in the early morning. When you go out to get in your car for work this morning, your friends come to ask you about the siren. However, you dont have enough time since youre in a hurry to catch your morning flight or you will be late. How will you excuse yourself from conversation politely? Sample Response 1 Score of 12: Hey, Id love to talk with you about last night. But, girls, listen..umm..I am in a rush and will be totally late if I dont go right now. Lets do it this evening, all right? Bye! Score of 9: Hi all, I will be late, girls, can we talk about it later? In the afternoon, okay? see you! Score of 6: Hi, um, I also heard about it, but, can we discuss it later, in the afternoon probably.
Score of 3: My friends, I am in a hurry and I need to go now, I dont want to delay my flight. Sample Item 2 (You have 2 minutes to read and think and 1 minute to respond) It is a lunch break and your professor wants to meet with you. While you are walking to his office, you see your friend from another class. He steps closer to you to talk about the
tailgate this weekend. Since meeting with the professor is really important for your grade, you dont want to show up late. What will you say to your friend? Sample Response 2 Score of 12: Hi Sandy, I know what you want to talk about, the tailgate right. I would love to, but I need to meet Professor Brown right now. How about talking about it later? Bye! Score of 9: Oh hi Sandy, wanna talk about the tailgate later? Heres the deal. I wanna see Professor Brown first and then come talk to you. Alright? Bye!
Score of 6: Sandy, if you wanna talk about the tailgate, excuse me first. I need to meet and talk to Professor Brown. Score of 3: Sandy, I cant talk now, I have to meet with Professor Brown.
Expressing concern
1. Prompt Attribute (based on the Barry item from LTA C3.3): This conversational task will propose a situation in which a test-takers friend is engaging in potentially harmful behavior. The test-taker will be asked to express concern to this friend.
The test item will be a short written passage that establishes the social background of the two friends. First, it will be established that they were close in earlier years and still maintain friendly social contact. Next, the potentially harmful behavior will be introduced as one in which both friends used to partake. Then it will be established that the test-taker gave up this behavior while the other did not. There will be a recent example of the testtakers behavior that moves the test-taker to worry about the friends conduct. Finally, the test-taker will be asked to raise these concerns with the friend. Creating new items: The type of behavior can be modified according to severity, rarity, and social stigma. Increasing any of these factors will also raise the item difficulty. However, the type should be one that people might plausibly engage in repeatedly despite
its detrimental effects. Various types of addiction could fit here. In order to reduce the item difficulty, a briefer background prompt could be developed, so that test-takers would not have to build on the past as much in order to get full credit. 1. Response Attribute: The test-taker will read the written prompt and respond to the task in a conversation with the examiner. The test-taker should express concern about their friends potentially damaging behavior. A response should proceed from the established social relationship that was mentioned in the prompt. As such, the test-taker might refer to previous experiences that they and their friend have shared. They might imply that the two of them have had similar discussions in the past, and might express dismay that their friend has fallen back into those harmful old habits.
The response should make sense. As long as the point is made, grading should not focus on grammatical accuracy, but on whether or not the test-taker states their concern effectively. The examiner should supply the friends answers based on the test-takers own utterances. 1. Sample Item (you have 10 minutes to read, decide what you want to say, and carry out the conversation): You are midway through your junior year of college. You have a roommate, Barry, with whom you have been living since freshman year. Barry is a close friend. On weekends you go out with a group of friends to movies, concerts, and parties. You enjoy talking together and frequently have long, philosophical conversations.
In the past, both you and Barry have had problems with procrastinating. You often avoided starting assignments until the last possible date, and then you typically produced poor-quality work due to lack of time. The stress of urgent deadlines weighed heavily on your minds. During final exams, the two of you would stay up late cramming for each test because you hadn't prepared well enough earlier.
You have now managed to get your procrastination under control. These days, you normally finish tasks a day or two ahead of time. Your grades have improved, and you feel less anxious. Barry also seemed to improve for a while, but now he appears to be backsliding. He spends most of his time playing video games and hardly ever
opens a textbook. Last semester he just barely achieved a C average. Now, at the beginning of the spring term, he shows no sign of wanting to change. You are worried. Initiate a conversation with Barry in which you discuss his procrastination problem. 1. Sample Response Score of 12: Test-taker: Hey, Barry. I think we need to talk.
Examiner: Huh? Hang on, Im in the middle of a game.
Test-taker: Yeah, thats the problem. Youre always playing games and never doing any homework. Im worried youll fail your classes. Examiner: Fine, Ill pause the game. So you think Im online too much? Examiner: Yeah. Test-taker: Exactly. Remember how we both tried to stop procrastinating, and it went really well for a while? Examiner: Yeah. But its my life.
Test-taker: Thats true. But youre also my friend, and I wanted to let you know I was worried. Examiner: All right, why not? Score of 9: Test-taker: Hey, Barry.
Test-taker: Well, it seems like youre backsliding now. You might have to take calculus twice just because you werent studying enough. You know what I mean?
Examiner: Well, thanks, I guess.
Test-taker: Sure. Ill be going to the library tonight if you want to come.
Examiner: Yeah, maybe.
Examiner: Yeah. But its my life.
Examiner: Really?
Examiner: All right. Whats up?
Examiner: Huh? Hang on, Im in the middle of a game. Test-taker: Okay, Ill wait.
Test-taker: So I want you to study more. Examiner: Okay, well see. Score of 6:
Test-taker: I used to have trouble with this too, but now its easier for me because I spend more time studying. Examiner: Yeah, I remember.
Test-taker: I think it will be hard for you to do well in school if you dont focus more.
Test-taker: Yeah. Its procrastination.
Test-taker: Um, I think youre playing too many computer games.
Test-taker: Excuse me, Barry?
Examiner: Huh? Hang on, Im in the middle of a game. Test-taker: No, I have to talk to you. Examiner: Fine, Ill pause the game. Examiner: Oh, really? Test-taker: Yeah. Its a problem. Examiner: Yeah. But its my life. Test-taker: Um, I guess. Test-taker: I think youre spending too much time on your computer.
Examiner: So?
Test-taker: I just think youre spending too much time on it. Examiner: All right. Score of 3:
Test-taker: Excuse me. Test-taker: Okay.
Test-taker: Youre not paying attention. Examiner: Is there something you wanted to tell me? Test-taker: Um, I dont know. Examiner: What? Sure I am. Test-taker: Okay.
Examiner: Fine, Ill pause the game.
Examiner: Huh? Hang on, Im in the middle of a game.
References:
Fulcher, G., and F. Davidson. (2007). Language Testing and Assessment: An Advanced Resource Book. London, UK: Routledge Press.
http://ed.fullerton.edu/sped/documents/SPEDApp/Oral_Interview_Rubric.pdf http://www.scribd.com/doc/95899066/Article-3
Amending the SAT for English Language Learners (ELLs), Version 2 Wahru (Esti) Sumarno, Jian Tao, and Erin Lamboi
SAT Amendment for ELLs The SAT is a standardized test used in determining college admission into United States universities and colleges. Over the past 75 years, the SAT has undergone many changes, not only in respects to test design but more so in the demographics of the test takers. However, surprisingly enough, modifications in the SAT have not reflected these particular changes. This test is known for the oarsmanregatta analogy question, which demonstrates why the test has been accused of cultural bias towards the white and the wealthy. With a growing number of English language learners (ELLs) coming from different parts of the world to take the SAT, the need to address the various issues involved with such a heterogeneous group, with their wide ranging ethnicities, cultural backgrounds, and native languages, becomes increasingly apparent. This issue becomes even more pressing when validity and reliability are taken into account.
Background: The Changing Population of SAT Takers As illustrated in the chart below, dramatic changes have occurred to the makeup of the test takers resulting in more diversified racial/ethnic backgrounds that the test makers have to take into account. The steepest increases in test volume since 1973 have been among students of Asian or Hispanic/Latino descent; the proportion of African American test-takers has also increased. Both groups together made-up one fifth of all SAT testtakers in 2001. However, the proportion of white test-takers decreased significantly from 87 percent in 1973 to 66 percent in 2001 (Lawrence et al, 2002). Over the years, these
changes in the ethnic backgrounds of test takers have had serious implications for test makers who share different first languages, cultures and basic knowledge of the world. In upholding the views of the validity theory, which should be the goal of not only SAT test makers but all test makers alike, there should be a focus on maintaining a certain degree of flexibility, which will allow for modifications to be made in response to the changes in our society. However, this is not the case.
A Brief Review of Major Changes in the SAT Studying the history of the SAT, how it has evolved, why it has evolved, and to what extent it has evolved or been modified constitutes a great resource for future SAT test makers to use in order to create a more reliable, more valid exam for the future. One of the major changes in the SAT has been that of time; the time allotted to each particular section of the test has gradually increased over the years. "Early versions of the SAT were quite speededas late as 1943, students were told that they should not expect to finish. Even so, many of the earlier modifications to the test aimed at providing more liberal time limits. In 1928 the test was reduced to seven subtests administered in 115
minutes, and in 1929, to six subtests" (College Board). Then, more recently, the SAT has been modified even further to five rather than six questions; the antonym format was changed to the more familiar five-choice question (College Board)." These modifications were meant to make the test more reliable, and ensure that what the test was created to evaluate was actually what was being tested. Despite the changes that have been made over the years, there is still much more work that needs to be done. More specifically, the needs of ELLs who are taking the test in the medium of their second or foreign languages need to be addressed when creating the SAT in order to administer a fair, unbiased, and reliable test. To narrow down our scope, this project will focus on sentence completion in the critical reading section, the most challenging part of the test.
Spec of Sentence Completion in the Current SAT 1. General Description: The SAT is a standardized test used by colleges and universities in the United States to assist in the admission process. The purpose of the test is to assess the critical thinking skills youll need for academic success in collegeand how well you apply what youve learned in
school to analyze and solve problems, the way youll need to in college" (SAT official guide, pp.3). It includes critical reading, mathematics and writing sections. The critical reading section includes reading passages and sentence completion. The writing section includes a short essay and multiple-choice questions on identifying errors and
improving grammar and usage. The mathematics section includes questions on arithmetic operations, algebra, geometry, statistics and probability. The Critical Reading (formerly Verbal) section of the SAT is made up of three scored sections: two 25-minute sections and one 20-minute section, with varying types of questions, including sentence completions and questions about short and long reading passages. Critical Reading sections normally begin with 5 to 8 sentence completion questions; the remainder of the questions are focused on the reading passages. Sentence completions generally test the student's vocabulary and understanding of sentence structure and organization by requiring the student to select one or two words that best complete a given sentence. Questions on the SAT reading-sentence completion are all multiple-choice questions which have five answer choices, only one of which is correct. The questions are weighted equally. For each correct answer, one raw point is added. For each incorrect answer onefourth of a point is deducted.
2. Prompt Attributes Sentence completion is administered by means of multiple-choice questions.. Test takers are given an incomplete sentence, with one to two missing words, that they must complete using the choices provided. The objective is to select the best choice among the five possible answers. Students have to do close reading and grasp a clear understanding of the sentence structure, content and basic background knowledge associated with the question.
3. Response Attributes
The test takers will be required to read and compare the five choices presented in the test item, and select best that completes the sentence, both in meaning and in form.
IV. Critique of Current SAT
1). Language Issue: The recondite vocabulary presented in the current test items prevents the test from assessing what the SAT is supposed to test. As mentioned above, the test is designed to evaluate students critical thinking and problem solving skills. Yet, in reality, the daunting vocabulary redirects the test to be more of a language test than an assessment of critical thinking and problem solving skills. As the number of ESL learners taking the SAT increases so does the responsibility of test makers to create a test that actually assesses critical thinking and problem solving skills. Being a language test, consisting of "vocabulary-loaded" questions, render huge obstacles for ESL test takers and does not fairly nor appropriately demonstrate how well they have truly mastered the target skills.
2). Cultural Issue: The prevalence of cultural bias has been a consistent problem in regards to the content of the test questions. Over the past decades, the major accusations against the test makers and the test itself have been related to cultural bias, specifically, towards wealthy, Caucasian Americans. The best example of this is the famous oarsmanregatta analogy question previously mentioned. Considering the fact that currently the test is moving beyond the United States and targeting students all around the world it only seems appropriate that the test should
also adjust to these changes. The SAT's almost almost exclusive emphasis on western culture, politics and history places certain test takers at an unfair advantage over others.
3). Time Issue: It is well known that we need more time to process information in our L2 than than the L1, especially when the two languages vary so greatly as does English and non-alphabet languages (Asian languages). The fact that Asian students are among the largest group of international SAT test takers, it is obvious that certain modifications need to be made in order to meet the needs of these "new" test takers and ensure that what the SAT aims to test is being adequately and appropriately met. Reforms to the test are imperative if validity is to be upheld. Furthermore, research has shown that test-takers relying solely on the "systematic" method of test taking, in which each answer choice is critically analyzed by the student were at a disadvantage due to time restraints. On the other hand, those test takers who depended on the Holistic approach were at an advantage because they simply had to look at the question, which was already familiar and easily recognized what choices were wrong, eliminating them immediately.
4). Test Fairness: Despite the growing number of international students taking the SAT, the test centers in the given region remain extremely limited in a way that exclude a large number of students who wish to go to an American college or university. China can be regarded as the most extreme example of this. Every year, thousands of Chinese high-school students have to travel all the way to Hong Kong, stay in a hotel and then try to find their way to the test center in a large, unfamiliar city. Apart from the limited access to the test, choosing computer-based over paper-based test is also
placing many students, unfamiliar with the computer, at a disadvantage. Both the scarcity of test centers and being computer-based only, excludes students from poor and middle class families from taking the SAT, all of which cast doubt on the fairness of the test.
V. Amendment to the Current SAT Spec: General Description The revised version of the SAT for ESL learners, will continue to be a standardized test used by colleges and universities in the United States to assist in the admissions process. The objectives will remain the same which are to "assess critical thinking skills" of high school students entering into college through testing math, writing, and critical reading skills." One mode of assessing critical reading skills, in specific, is through sentence completion, which is the focus of this spec. The questions of the revised version will be similar to the original with a few modifications made in respects to the criticism mentioned above. The word choices will not change and the level of difficulty will remain the same. However, the revised version will allow the test takers to use an English dictionary. The dictionary will be provided during testing and will be available to ESL test takers as well as native English test takers. In regards to cultural bias, the revised SAT examples will exclude all cultural references. The test questions will not longer be dominated by references to Western Society. Rather, questions will represent fictional countries, cultures, and ethnicities, to ensure fairness to all test takers.
With respect to time, more time should be allotted to ESL SAT takers to compensate for the extra time spent manipulating a language other than their own as well as factors such as jet lag and culture shock. Offering the test over two days instead of one, offering more breaks during testing, and allowing more time overall to take the test. In this way speed does not become an issue like it had become in the past before the test was modified, offering more time and including less questions. The Revised SAT will also be made available in more countries and more accessible within each of these countries by offering the test outside of the cities as well as within. The test will also be offered in both paper-based and computer-baed forms depending on each areas preference.
2. Prompt Attributes Sentence completion is administered by means of multiple-choice questions. Test takers are given an incomplete sentence, with one to two missing words, that they must complete using the choices provided. The objective is to select the best choice among the five possible answers. In order to put test takers of different cultures, races and ethnicities at a fair platform, the test items should avoid social, cultural and historical issues that might facilitate problem solving. Instead, fictional scenarios will be created when needed. In addition, the vocabulary remains the same provided in the original SAT exam, however, in the revised version, a handy dictionary will be offered to every test taker. The test should allot more time to ESL learners to complete the test and offer longer breaks during testing.
3. Response Attributes The test taker will first read the prompt and review the five answer choices. Then reread the prompts and answer choices critically, through an analysis of sentence structure, context and meaning, in order to ensure complete understanding of what is being asked, then predict the most appropriate word to complete the sentence item. Through the process of elimination the test taker will eliminate answers they deem incorrect until the correct choice is decided upon.
4. Sample Items Original Sample Item 1: Because King Philips desire to make Spain to dominant power in sixteenth-century
Europe ran counter to Queen Elizabeths insistence on autonomy for English, _____was_____. A. reconciliationassured
B. warfareavoidable
C. ruinationimpossible
D. conflictinevitable
E. diplomacysimple Revised Sample Item 1:
Because King Erins desire to make Lamboiville to dominant power in sixteenth-century
Europe ran counter to Queen Tracys insistence on autonomy for Taoville, _____was_____. reconciliationassured
warfareavoidable
ruinationimpossible
conflictinevitable
diplomacysimple
diplomacysimple Reason of Revision: Since the test is targeting students all around the world, it should not only involve questions related to Western history. However, it would be impossible to take into account all the different countries, cultures, and ethnicities of the world. Therefore, the revised version of the SAT will use fictional examples.
Original Sample Item 2: There is no doubt that Larry is a genuine_____: he excels at telling stories that fascinate his listeners. braggart (71) dilettante (118)
pilferer (19) prevaricator (11) raconteur (97)
Reason of Revision: As mentioned above, test contains numerous uncommon words making it more of a language test to native speakers, not even mention EFL/ESL learners. For example, we searched the five choices in the Corpus of Contemporary American English (COCA) (450 million words in total) and came up with the number of tokens listed on the right side of the corresponding word. Obviously, choice (c) and (d) are extremely infrequent used. Then there seems little justification to test a word that students will rarely use, ESL test takers as well as native test takers. Therefore we have provided a dictionary for both native speakers and ESL learners to use during testing.

Fall 2012 Spec Packet

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Fall 2012 Spec Packet

Caricato da

Copyright:

Formati disponibili

Fall 2012 EIL 460 Test Specification Packet

Understanding Katie Honken, Version Version Version

Intermediate Oral Grammar Achievement Test V1.0

Spec Packet, EIL 460, Fall 2012, page 1

Spec Packet, EIL 460, Fall 2012, page 2

Spec Packet, EIL 460, Fall 2012, page 3

Spec Packet, EIL 460, Fall 2012, page 4

Spec Packet, EIL 460, Fall 2012, page 5

Spec Packet, EIL 460, Fall 2012, page 6

Oral Warm-Up Interview

Spec Packet, EIL 460, Fall 2012, page 7

Spec Packet, EIL 460, Fall 2012, page 8

Task 2: Simple Past

Spec Packet, EIL 460, Fall 2012, page 9

Geography Class Chemistry Class Soccer Practice Study

__________ Class Math Class Dinner

8:30 PM 12:00 AM 12:00-1:30 AM

Spec Packet, EIL 460, Fall 2012, page 10

Geography Class Chemistry Class Soccer Practice Dinner Study

11:00 AM 1:00 PM 1:00-2:00 PM 2:00-3:30 PM 8:30 PM 12:00 AM 12:00-1:30 AM 7:30-8:30 PM

Math Class Dinner Study

Spec Packet, EIL 460, Fall 2012, page 11

Task 3: Modals (Possibility)

Spec Packet, EIL 460, Fall 2012, page 12

Task 4: Present Progressive

(Communicative Grammar Practice, Leo Jones, 1985)

Spec Packet, EIL 460, Fall 2012, page 13

Spec Packet, EIL 460, Fall 2012, page 14

Task 5: Present Progressive

Spec Packet, EIL 460, Fall 2012, page 15

(Azar Beginner Grammar Textbook Series)

Spec Packet, EIL 460, Fall 2012, page 16

Spec Packet, EIL 460, Fall 2012, page 17

Task 6: Modals (Necessity)

Spec Packet, EIL 460, Fall 2012, page 18

Task 7: Articles / Quantifiers

Spec Packet, EIL 460, Fall 2012, page 19

Spec Packet, EIL 460, Fall 2012, page 20

Spec Packet, EIL 460, Fall 2012, page 21

Simple Present WH- Questions

Simple Past Possibility Modals

Task 5: Past Progressive Task 6: Necessity Modals Task 7: Articles

Spec Packet, EIL 460, Fall 2012, page 22

Spec Packet, EIL 460, Fall 2012, page 23

Objective #1: Assessing students ability to understand written directions of assignment

Spec Packet, EIL 460, Fall 2012, page 24

Spec Packet, EIL 460, Fall 2012, page 25

Spec Packet, EIL 460, Fall 2012, page 26

Item 2: Writing correct file names

Spec Packet, EIL 460, Fall 2012, page 27

Spec Packet, EIL 460, Fall 2012, page 28

Spec Packet, EIL 460, Fall 2012, page 29

Objective #3: Assessing students ability to understand and apply a rubric.

Spec Packet, EIL 460, Fall 2012, page 30

Formatting The paragraph meets The paragraph

The paragraph is at least 85 words long.

The paragraph is 75 The paragraph is 65 to 84 words long. to 74 words long.

Spec Packet, EIL 460, Fall 2012, page 31

Spec Packet, EIL 460, Fall 2012, page 32

Objective #4: Assessing students ability to understand written feedback

Spec Packet, EIL 460, Fall 2012, page 33

Spec Packet, EIL 460, Fall 2012, page 34

Spec Packet, EIL 460, Fall 2012, page 35

Spec Packet, EIL 460, Fall 2012, page 36