Vicky Fang MATESOL

Component 8: Assessment Project 1 Purpose and Audience

As a language assessment project with Dr. Kathleen M. Bailey, I worked with Hala Sun, who is a Dordt College alumna, to re-design the Dordt College Placement Test (DCPT). The test was created 16 years ago by the current English for Academic Purpose (EAP) course instructor, Sanneke Kok. All students admitted to Dordt College for whom English is not their native language are required to take the DCPT, which intends to measure test-takers academic English proficiency in listening, reading, writing, and speaking skills as well as their grammatical knowledge. If the students do not pass this test, they will be required to take either one of the EAP courses or to take both EAP coursesAcademic Writing and/or Academic Interaction based on their scores on each part of the test. Constructs and Organization With Alderson, Clapham, and Walls (1995) guidance on developing test specifications, we identified the five constructs for the new placement test. There are listening comprehension, grammatical knowledge, reading comprehension, writing ability and oral skills. We used integrative and direct tests to measure each construct because the direct and integrative test methods are better to predict students use of the target language in real life (Bailey, 1998; Wesche, 1983). We constructed and designed the test as follows (see Appendix A for the complete placement test). In defining the listening construct, Buck (2001) argues that listening tests need to be contextualized, knowledge-independent, require fast, automatic, on-line processing of texts and go beyond literal meaning (p. 113). In correspondence to the definition, we made a listening task that requires test-takers to respond to five short-answer questions after watching a four-minute video. By doing so, we simulated a mini lecture to test students abilities to recall

Vicky Fang MATESOL

Component 8: Assessment Project 2

specific words as well as students comprehension of the overall speech. From our interview with Instructor Kok, we learned that the institutions educational philosophy stresses an emphasis on students grammatical competence. Therefore, we added the grammar section in designing the test. Citing Larsen-Freeman (1991, 1997), Brown (2010) also argues that grammatical knowledge includes grammatical forms, grammatical meanings and pragmatic meaning. To implement the idea that grammatical forms are intimately associated with grammatical meanings as well as pragmatic meaning and thus should be contextualized, we used the cloze-elide test technique (Alderson, 2000). After selecting a text, we identified and inserted 15 common grammar errors that ESL learners may make in the text. Test-takers need to cross out the 15 intrusive words that make the sentences grammatically incorrect. Hedgcock and Ferris (2009) mention that from a bottom-up view of reading, the reader starts from small units such as words and works towards large units such as written discourse; from the top-down view, the readers understanding of a text is the product of the readers background knowledge of the text and the information given by the text. Thus, we designed the items that lead the test-takers to adopt both approaches to comprehend the reading passage (Alderson, 2000). The bottom-up items include questions asking for the interpretation of specific words. The top-down items include questions that require the test-takers to paraphrase a sentence and recognize the implied message of a text. We included two readings in the section, which consists of 10 multiple-choice questions. To study academic writing, students need to master the process of structuring ideas into a piece of writing which shares the convention of a specific type of text (Ferris & Hedgcock, 1998). To measure the students writing skills, we decided to assess the students ability to write an expository essay, which is a common essay genre that college students often encounter in

Vicky Fang MATESOL

Component 8: Assessment Project 3

academic life (Purdue Online Writing Lab, 2010). Test-takers need to write an essay of about 180-250 words to state, explain, and support their views on the given prompt. Luoma (2004) defines speaking tasks as activities that involve speakers in using language for the purpose of achieving a particular goal or objective in a particular speaking situation (p. 31). To effectively assess the construct, we created a prompt that requires testtakers to expound on an argument based on a given topic. Test-takers will have two minutes to prepare their speech and three minutes to perform their speech orally. During the preparation time, test takers are also allowed to jot down some notes for their speech. Scoring Criteria Reading comprehension and grammar are both objectively scored subtests. For reading comprehension, each multiple-choice item is worth one point. For the grammar subtest, testtakers only receive points when they cross out the extra words in the article. We used the exact word method to evaluate responses that require specific information from the listening, and we used the acceptable word method to assess the test-takers comprehension of the overall content of the listening (Bailey, 1998). Each question is worth two points. One point will be deducted if the test-taker does not respond to the acceptable word questions in a complete sentence. Using two analytical rubrics, we assign the writing section 100 points and the oral section 40 points (see Appendix B). In responding to Dordt Colleges focus on developing students grammatical competence, the grammar part of the writing rubric weighs 50% of the total possible writing scores.

Vicky Fang MATESOL References

Component 8: Assessment Project 4

Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press. Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press. Angeli, E., Wagner, J., Lawrick, E., Moore, K., Anderson, M., Soderlund, L., & Brizee, A. (2010, May 5). Expository essays. Retrieved from http://owl.english.purdue.edu/owl/resource/685/02/ Bailey, K. M. (1998). Learning about language assessment: Dilemmas, decisions, and directions. Boston, MA: Heinle & Heinle Publishers. Brown, H. D. (2010). Language assessment, principles and classroom practice. New York, NY: Pearson Education. Buck, G. (2001). Assessing listening. Cambridge: Cambridge University Press. Ferris, D.R., & Hedgcock, J.S. (1998). Teaching ESL composition: Purpose, process, and practice. Mahwah, NJ: Lawrence Erlbaum Associates. Hedgcock, J. S., & Ferris, D. R. (2009). Teaching readers of English: Students, texts, and contexts. New York, NY: Routledge. Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge University Press. Wesche, M. B. (1983). Communicative testing in a second language. Modern Language Journal, 67, 4155.

Vicky Fang MATESOL

Component 8: Assessment Project 5

Appendix A Dordt College Placement Test (DCPT)

Instruction: The placement test consists of 5 sections (about 1 hour total) For the listening comprehension, the test instructor will play a short video. I. Listening Comprehension (10 minutes) II. Grammar (5 minutes) III. Reading Comprehension (20 minutes) IV. Mini-Essay Writing (20 minutes) After you complete all four sections, submit your test to the test instructor and schedule a time to do the oral interview section. The test instructor will provide you the oral interview section of the test. You will be interviewed individually and the interview will be audiorecorded. V. Oral Interview (about 5 minutes)

Name: ___________________

Vicky Fang MATESOL I. Listening Comprehension: Short-Answer Questions

Component 8: Assessment Project 6

Watch the video Worlds English Mania presented by Jay Walker from Ted Talk (about 4 minutes).1 Use the space below to take notes. After watching the video, answer the following five questions. Please answer in less than 50 words (ALL ANSWERS MUST BE IN COMPLETE SENTENCES EXCEPT FOR QUESTIONS 2 & 3) Use this space to take notes

TURN TO NEXT PAGE FOR QUESTIONS

Walker, J. (2009, February). Jay Walker: Worlds English mania [Video file]. Retrieved from http://www.ted.com/talks/lang/en/jay_walker_on_the_world_s_english_mania.html
1

Vicky Fang MATESOL

Component 8: Assessment Project 7

Please answer in less than 50 words (ALL ANSWERS MUST BE IN COMPLETE SENTENCES EXCEPT FOR QUESTIONS 2 & 3)

1. In your own words, define the word mania.

2. How many people are trying to learn English worldwide?

3. Name at least 3 places (countries or regions) that the speaker mentioned that HAVE manias for English?

4. According to the speaker, why are so many people trying to learn English?

Vicky Fang MATESOL

Component 8: Assessment Project 8

5. What is the speakers opinion of English manias?

Vicky Fang MATESOL

Component 8: Assessment Project 9

II. Grammar: The passage was taken from the New York Times newspaper, published on April 3, 2012.2 Read the following passage and cross out 15 extra words that make the sentences grammatically incorrect. Example: The boys is are singing the national anthem. How Immersion Helps to Learn a Language Learning a the foreign language is never easy, but contrary to common wisdom, it is possible for adults to process a language a the same way a the native speaker does. And over time, processing improves even when the skill goes unused, researchers are reporting. For there their study, in on the journal PloS One, the scientists used an artificial language of 13 words, completely different from English. Its totally unpractical impractical to follow someone to high proficiency because it takes years and years, said the lead author, Michael Ullman, a neuroscientist at Georgetown University Medical Center. The language dealt with pieces and moves in a the computer game, and the researchers tested proficiency by asking test subjects to play a the game. The subjects are were split into two groups. One group studied the language in a formal classroom setting, while the other was were trained through immersion. After five months, both groups retained the language even though because they had not used it at all, and both displayed brain processing similar to that of a native speaker. But the immersion group displayed the full brain patterns for of a native speaker, Dr. Ullman said. The research has several applications, Dr. Ullman said. This should help us understand how foreign-language learners can achieve native like processing with increase increased practice, he said. It makes sense that youd want to have your brain process like a the native speaker. And though it may take takes time, and more research, the work also could or should help in rehabilitation of people with traumatic brain injury, he added.

Bahanoo, S. (2012, April 3). How immersion helps to learn a new language. The New York Times. Retrieved from http://www.nytimes.com/2012/04/03/science/how-immersion-helps-to-learn-a-new-language.html

Vicky Fang MATESOL III. Reading Comprehension:

Component 8: Assessment Project 10

Passage 1: The passage was taken from the National Geographic Learning.3 Read the passage below and answer the multiple-choice questions following the passage. Circle the letter of the best answer. The Worlds Oldest First Grader On January 12, 2004, Kimani Maruge knocked on the door of the primary school in his village in Kenya. It was the first day of school, and he was ready to start learning. The teacher let him in and gave him a desk. The new student sat down with the rest of the first graderssix- and seven-year-old boys and girls. However, Kimani Maruge was not an ordinary first grader. He was 84 years oldthe worlds oldest first grader. Kimani Maruge was born in Kenya in 1920. At that time, primary education in Kenya was not free, and Maruges family didnt have enough money to pay for school. When Maruge grew up, he worked hard as a farmer. In the 1950s, he fought with other Kenyans against the British colonists. After years of fighting, Kenya became independent in 1963. In 2003, the Kenyan government began offering free primary education to everyone, and Maruge wanted an education, too. However, it wasnt always easy for Maruge to attend school. Many of the first graders parents didnt want an old man in their childrens class. School officials said that a primary education was only for children. But the school principal, Jane Obinchu, believed Maruge was right. With her help, he was able to stay in school. Maruge was a motivated and successful student. In fact, he was one of the top five students in his first grade class. In second grade, Maruge became the schools student leader. He went as far as seventh grade, the final year of primary school. Over the years, Maruge studied Swahili, English, and math. He wanted to use his education to read the Bible and to study veterinary medicine. In 2008, there were problems in Kenya after an election. People were fighting and burning houses in Maruges village. Maruge moved to a refugee camp for safety and lived in a tent. However, even during those difficult times he continued to go to school. Later that year, he moved to a home for the elderly. He continued going to school, and even taught other residents of the home to read and write. In 2005, Maruge flew in a plane for the first time in his life. He traveled to New York City, where he gave a speech at the United Nations. He spoke about the importance of education and asked for help to educate the people of Kenya. Maruge also wanted to improve primary education for children in Africa. Maruge died in 2009, at age 89. However, his story lives on. The 2010 movie The First

The passage was printed in Vargo, M. & Blass, L. (2013). Pathways 1: Reading, writing, and critical thinking. G National Geographic Learning. Boston:

Vicky Fang MATESOL

Component 8: Assessment Project 11

Grader showed Maruges amazing fight to get an education. Many older Kenyans decided to start school after seeing The First Grader. One of those people was 19-yearold Thoma Litei. Litei said, I knew it was not too late. I wanted to read, and to know more language, so I came [to school] to learn. That is why it is important for his story to be known. 1. Based on the passage, we can infer that before 2003, primary education in Kenya was: a. Not cheap b. Not available c. Prohibited d. Free 2. Why was Maruge motivated to study? a. To be in one of the top five students in his class. b. To use his education to read the Bible. c. To become the schools student leader. d. To study Swahili, English, and math. 3. Who did NOT want Maruge to be in school? a. Kenyan government b. First grade parents c. Jane Obinchu d. None of the above 4. The main idea in paragraph (E) is: a. People were fighting and burning houses in the village. b. It was too difficult to live in a tent at a refugee camp. c. Maruge did not stop studying, even during those difficult times. d. Maruge taught other residents of the home to read and write. 5. The main idea in paragraph (G) is: a. Maruge was an inspiration to other adult Kenyans. b. Kenyans enjoyed the movie The First Grader.

Vicky Fang MATESOL c. Thoma Litei decided to go to school to learn. d. The First Grader was created after Maruges death.

Component 8: Assessment Project 12

Passage 2: The following extract was taken from the article Does Your Language Shape How You Think? published in the New York Times magazine4. Read the passage below and answer the multiple-choice questions following the passage. Circle the letter of the best answer. Benjamin Lee Whorfs theory crash-landed on hard facts and solid common sense, when it transpired1 that there had never actually been any evidence to support his fantastic claims. The reaction was so severe that for decades, any attempts to explore the influence of the mother tongue on our thoughts were relegated2 to the loony3 fringes4 of disrepute5. But 70 years on, it is surely time to put the trauma of Whorf behind us. And in the last few years, new research has revealed that when we learn our mother tongue, we do after all acquire certain habits of thought that shape our experience in significant and often surprising ways. Vocabulary word-bank: 1. transpire: occur, happen 4. fringe: border, trimming

2. relegate: assign, transfer 5. disrepute: dishonor

3. loony: crazy

1. The authors attitude to Whorfs theory is a. Ambivalent b. Neutral c. Supportive d. Contemptuous 2. The word trauma in the passage is closest in meaning to a. Physical injury b. Torture c. Emergency d. Agony
4

Deutscher, G. (2010, August 26). Does your language share how you think? The New York Times. Retrieved from http://www.nytimes.com/2010/08/29/magazine/29language-t.html?pagewanted=all

Vicky Fang MATESOL

Component 8: Assessment Project 13

3. All of the following can be inferred from the text EXCEPT a. Learning our mother tongue can lead to positive experiences. b. The influence of mother tongue on our thoughts is significant. c. Whorfs theory was based on hard facts and solid common sense. d. Whorf failed to provide any evidence to support his theory. Turn to the next page 4. The author uses the word crash-landed to imply that Whorfs theory was _________ hard facts and solid common sense. a. in favor of b. based on c. inconsistent with d. critical of 5. Which of the sentences below best expresses the essential information in the boldfaced sentence in the passage? a. Exploring the relationship between the mother tongue and our thoughts was frowned upon for decades. b. People reacted severely and they explored the relationship between the mother tongue and our thoughts. c. Whorfs theory succeeded in exploring the relationship between the mother tongue and our thoughts. d. Whorfs claims were so credible that no researcher made an attempt to dishonor Whorf for decades.

Vicky Fang MATESOL

Component 8: Assessment Project 14

IV. Mini-essay writing Write a mini-essay about 180-250 words according to the following prompt. You will be tested on the following criteria: content, organization, and grammar. Do you think learning English is important? If so, why or why not? Please provide personal examples to support your stance (in addition, you may refer to what you have learned from the video English Mania). _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _

Vicky Fang MATESOL

Component 8: Assessment Project 15

_____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _ _____________________________________________________________________________ _

Vicky Fang MATESOL

Component 8: Assessment Project 16

_____________________________________________________________________________ _ _____________________________________________________________________________ _

END OF SECTION IV SUBMIT YOUR TEST AND SCHEDULE AN ORAL INTERVIEW

NAME:_________________ V. Oral Interview: In the United States, many universities require students to learn an additional language other than their native language. Do you think universities in your home country should require students to learn an additional language (other than your native language)? Why or why not? You have 2 minutes to prepare. Use the space below to write down an outline or important points that you want to discuss. You will be given maximum 3 minutes to answer the question. You can use your notes to talk but do not read aloud what you have written out. Please relate the issue to your personal experience and cultural background.

Vicky Fang MATESOL

Component 8: Assessment Project 17

Appendix B Analytic Rubrics Table B1 Mini-essay writing Scoring Criteria Content Scoring: circle the appropriate score Clearly relates or answers to the given Clear 543210 Missing topic or question Gives sufficient examples/references Sufficient 543210 Lacking Clear connection between Clear 543210 Missing examples/references and main ideas Correct use of vocabulary words Correct 543210 Incorrect Sufficient number of words (180-250) Target #: 5; 160-179 words: 4; 140-159 words: 3; 120139 words: 2; 100-119 words: 1; less than 100 words: 0 Subtotal: points for content _________/25 Organization Scoring: circle the appropriate score Topic or introductory sentence Clear 54Not Clear 321Missing 0 Concluding sentence Clear 54Not Clear 321Missing 0 Coherence (logical progression and Always 54Sometimes 321 Never 0 development of ideas, good flow) Cohesion (good connections between Always 54Sometimes 321Never 0 sentences) Sentence variety (both simple and Good Variety 54Some Variety 321__Never 0 compound and/or complex) Subtotal: points for organization ________/25 Grammar Scoring: take off one point for each error in the categories indicated. Circle the # of remaining pts. Correct spelling (subtract 1 pt .ea. new 5 4 3 2 1 0 error) Correct use of articles and prepositions 5 4 3 2 1 0 Standard capitalization 5 4 3 2 1 0 Standard punctuation (periods, commas, 5 4 3 2 1 0 semicolons) Standard sentence word order 5 4 3 2 1 0 Agreement between subjects & verbs, 5 4 3 2 1 0 nouns and pronouns/antecedents

Vicky Fang MATESOL

Component 8: Assessment Project 18 5 5 5 5 4 4 4 4 3 3 3 3 2 2 2 2 1 1 1 1 0 0 0 0

Correct verb tense and usage Correct adverb and adjective usage Appropriately placed phrasal modifiers Standard academic diction (avoidance of slang and informal language) Subtotal: points for grammar TOTAL POINTS

_______/50 _______/100 Table B2 Oral Interview Criteria Scoring: circle the appropriate score Clear 543210 Missing

Content Clearly relates or answers to the given topic or question

Gives adequate and meaningful examples/references Sufficient 543210 Lacking Clear connection between examples/references and main ideas Correct use of vocabulary words Accuracy Correct use of grammar Clear pronunciation of words Fluency Coherence (logical progression and development of Always 54Sometimes 321 0 ideas, good flow) Never Fluency in speech (with few use of circumlocution and few hesitation) TOTAL POINTS Fluent 54 Somewhat Fluent 321 0 Not Fluent ______/40 Correct 543210 Incorrect Clear 54Not Clear 3210 Missing Clear 543210 Missing Correct 543210 Incorrect

Vicky Fang MATESOL

Component 8: Assessment Project 19