Assignment 5

LANGUAGE TESTS

The previous chapters introduced a number of building blocks for designing language tests. You now have a sense of where tests belong in the larger domain of assessment. You have sorted through differences between formal and informal tests. formative and summative tests, and norm- and criterion-referenced tests. You have traced some of the historical lines of thought in the field of language assessment. You have a sense of major current trends in language assessment, especially the present focus on communicative and process-oriented testing that seeks to transform tests from anguishing ordeals into challenging and intrinsically motivating learning experiences. By now, certain foundational principles have entered your vocabulary: practicality, reliability, validity, authenticity, and washback. And you should now possess a few tools with which you can evaluate the effectiveness of a classroom test.

TEST TYPES

The first task you will face in designing a test for your students is to determine the purpose for the test. Defining your purpose will help you choose the right kind of test, and it will also help you to focus on the specific objectives of the test. We will look first at two test types that you will probably not have many opportunities to create as a classroom teacher-language aptitude tests and language proficiency tests-and three types that you will almost certainly need to create-placement tests, diagnostic tests, and achievement tests.

LANGUAGE APTITUDE TESTS

One type of test-although admittedly not a very common one-predicts a person's success prior to exposure to the second language. A language aptitude test is designed to measure capacity or general ability to learn a foreign language and ulti- mate success in that undertaking. Language aptitude tests are ostensibly designed to apply to the classroom learning of any language. Two standardized aptitude tests have been used in the United States: the Modern Language Aptitude Test (MLAT) (Carroll & Sapon, 1958) and the Pimsleur Language Aptitude Battery (PLAB) (Pimsleur, 1966). Both are English language tests and require students to perform a number of language-related tasks. The MLAT, for example, consists of five different tasks.

Tasks in the Modern Language Aptitude Test

1.      Number learning: Examinees must learn a set of numbers through aural input and then discriminate different combinations of those numbers.

2.      Phonetic script: Examinees must learn a set of correspondences between speech sounds and phonetic symbols.

3.      Spelling clues: Examinees must read words that are spelled somewhat phonetically, and then select from a list the one word whose meaning is closest to the disguised" word.

4.      Words in sentences: Examinees are given a key word in a sentence and are then asked to select a word in a second sentence that performs the same grammatical function as the key word.

5.      Paired associates: Examinees must quickly leam a set of vocabulary words from another language and memorize their English meanings.

A typical example of a standardized proficiency test is the Test of English as a Foreign Language (TOEFL.) produced by the Educational Testing Service. The TOEFL is used by more than a thousand institutions of higher education in the United States as an indicator of a prospective student's ability to undertake academic work in an English-speaking milieu. The TOEFL consists of sections on listening comprehension, structure (or grammatical accuracy).reading comprehension, and written expression. The new computer-scored TOEFL announced for 2005 will also include an oral production component. With the exception of its writing section, the TOEFL (as well as many other large-scale proficiency tests) is machine-scorable for rapid turnaround and cost effectiveness (that is, for reasons of practicality).

Research is in progress(Bernstein et al., 2000) to determine, through the technology of speech recognition, if oral production performance can be adequately machine-scored. (Chapter 4 provides a comprehensive look at the TOEFL and other standardized tests). A key issue in testing proficiency is how the constructs of language ability are specified. The tasks that test-takers are required to perform must be legitimate samples of English language use in a defined context. Creating these tasks and validating them with research is a time-consuming and costly process. Language teachers would be wise not to create an overall proficiency test on their own. A far more practical method is to choose one of a number of commercially available proficiency tests.

PROFICIENCY TESTS

If your aim is to test global competence in a language, then you are, in conventional terminology, testing proficiency A proficiency test is not limited to any one course, curriculum, or single skill in the language; rather, it tests overall ability. Proficiency tests have traditionally consisted of standardized multiple-choice items on grammar, vocabulary, reading comprehension, and aural comprehension. Sometimes a sample of writing is added, and more recent tests also include oral production performance. As noted in the previous chapter, such tests often have content validity weaknesses, but several decades of construct validation research have brought us much closer to constructing successful communicative proficiency tests.

PLACEMENT TESTS

Certain proficiency tests can act in the role of placement tests, the purpose of which is to place a student into a particular level or section of a language curriculum or school. A placement test usually, but not always, includes a sampling of the material to be covered in the various courses in a curriculum; a student's performance on the test should indicate the point at which the student will find material neither too easy nor too difficult but appropriately challenging. The English as a Second Language Placement Test (ESLPT) at San Francisco State University has three parts. In Part I. students read a short article and then write a summary essay. In Part II, students write a composition in response to an article, Part 111 is multiple-choice: students read an essay and identify grammar errors in it. The maximum time allowed for the test is three hours. Justification for this three part structure rests largely on the test's content validation. Most of the ESL courses at San Francisco State involve a combination of reading and writing, with a heavy emphasis on writing. The first part of the test acts as both a test of reading com- prehension and a test of writing (a summary). The second part requires students to state opinions and to back them up, a task that forms a major component of the writing courses.

DIAGNOSTIC TESTS

A diagnostic test is designed to diagnose specified aspects of a language. A test in pronunciation, for example, might diagnose the phonological features of English that are difficult for learners and should therefore become part of a curriculum. Usually, such tests offer a checklist of features for the administrator (often the teacher) to use in pinpointing difficulties. A writing diagnostic would elicit a writing sample from students that would allow the teacher to identify those rhetorical and linguistic features on which the course needed to focus special attention. Diagnostic and placement tests, as we have already implied, may sometimes be indistinguishable from each other.

ACHIEVEMENT TESTS

An achievement test is related directly to classroom lessons, units, or even a total curriculum. Achievement tests are or should be limited to particular material addressed in a curriculum within a particular time frame and are offered after a course has focused on the objectives in question. Achievement tests can also serve the diagnostic role of indicating what a student needs to continue to work on in the future, but the primary role of an achievement test is to determine whether course objectives have been met-and appropriate knowledge and skills acquired-by the end of a period of instruction. Achievement tests are often summative because they are administered at the end of a unit or term of study. They also play an important formative role. An effective achievement test will offer wash back about the quality of a learner's performance in subsets of the unit or course.

SOME PRACTICAL STEPS TO TEST CONSTRUCTION

The descriptions of types of tests in the preceding section are intended to help you understand how to answer the first question posed in this chapter: What is the purpose of the test? It is unlikely that you would be asked to design an aptitude test or a proficiency test, but for the purposes of interpreting those tests, it is important that you understand their nature. However, your opportunities to design placement, diagnostic, and achievement tests-especially the latter-will be plentiful, In the remainder of this chapter, we will explore the four remaining questions posed at the outset, and the focus will be on quipping you with the tools you need to create such classroom-oriented tests.

ASSESSING CLEAR, UNAMBIGUOUS OBJECTIVES

In addition to knowing the purpose of the test you're creating, you need to know as specifically as possible what it is you want to test. Sometimes teachers give tests simply because it's Friday of the third week of the course, and after hasty glances at the chapter(s) covered during those three weeks, they dash off some test items so that students will have something to do during the class. This is no way to approach a test. Instead, begin by taking a careful look at everything that you think your students should "know" or be able to "do," based on the material that the students are responsible for. In other words, examine the objectives for the unit you are testing.

Selected objectives for a unit in a low-intermediate integrated-skills course Form-focused objectives (listening and speaking)

1.      Students will   recognize and produce tag questions, with the correct grammatical form and final intonation pattern, in simple social conversations.

2.      recognize and produce what information questions with correct final intonation pattern.

3.      Communication skills (speaking)Students will state completed actions and events in a social conversation.

4.      Ask for confirmation in a social conversation.

5.      Give opinions about an event in a social conversation

6.      produce language with contextually appropriate intonation, stress, and rhythm.

7.      Reading skils (simple essay or story) Students will recognize iregular past tense of selected verbs in a story or essay.

8.      Writing skills (simple essay or story) Students will write a one-paragraph story about a simple event in the past.

9.      Use conjunctions so and because in a statement of opinion.

DRAWING UP TEST SPECIFICATIONS

Test specifications for classroom use can be a simple and practical outline of your test. (For arge-scale standardized tests [see Chapter 4] that are intended to be widely distributed and therefore are broadly generalized, test specifications are more formal and detailed.) In the unit discussed above, your specifications mu will simply comprise (a) a broad outline of the test, (b) what skills you will test, and ()what the items will look like. Let's look at the first two in relation to the midterm unit assessment already referred to above (a) Outline of the test and (b) skils to be included. Because of the constraints of your curriculum, your unit test must take no more than 30 minutes.

DEVISING TEST TASKS

Your oral interview comes first, and so you draft questions to conform to the accepted pattern of oral interviews (see Chapter 7 for information on constructing oral interviews). You begin and end with nonscored items (warm-up and wind-down) designed to set students at ease, and then sandwich between them items intended to test the objective (level check) and a little beyond (probe). Oral interview format

A. Warm-up: questions and comments

B. Level-check questions (objectives 3, 5, and6)

1. Tell me about what you did last weekend.

2. Tell me about an interesting trip you took in the last year.

3. How did you like the TV show we saw this week?

C. Probe (objectives 5, 6)

1. What is your opinion about

? (news event)

2. How do you feel about.

-? (another news event)

D. Wind-down: comments and reassurance

You are now ready to draft other test items. To provide a sense of authenticity and interest, you have decided to conform your items to the context of a recent TV sitcom that you used in class to illustrate certain discourse and form-focused factors.

DESIGNING MULTIPLE-CHOICE

Test Items In the sample achievement test above, two of the five components (both of the listening sections) specified a multiple-choice format for items. This was a bold step to take. Multiple-choice items, which may appear to be the simplest kind of item to construct, are extremely difficult to design correctly. Hughes (2003, pp. 76-78) cautions against a number of weaknesses of multiple-choice items: The technique tests only recognition knowledge. Guessing may have a considerable eftect on test scores. The technique severely restricts what can be tested. It is very difficult to write successful items. Washback may be harmful.

SCORING, GRADING, AND GIVING FEEDBACKK

Scoring as you design a classroom test, you must consider how the test will be scored and graded. Your scoring plan reflects the relative weight that you place on each section and items in each section. The integrated-skills class that we have been using as an example focuses on listening and speaking skills with some attention to reading and writing. Three of your nine objectives target reading and writing skills. How do you assign scoring to the various components of this test? Because oral production is a driving force in your overall objectives, you decide to place more weight on the speaking (oral interview) section than on the other three sections. Five minutes is actually a long time to spend in a one-on-one situation with a student, and some significant information can be extracted from such a session. You therefore designate 0 percent to the grade to the oral interview. 1ou consider the listening and reading sections to be equally important, but each of them, especially in this multiple-choice format, is of less consequence than the oral interview. So you give each of them a 20 percent weight.

That leaves 20 percent for the writing section, which seems about right to you given the time and focus on writing in this unit of the course. Your next task is to assign scoring for each item. This may take a little numerical common sense, but it doesn't require a degree in math. To make matters simple, you decide to have a 100-point test in which the listening and reading items are each worth 2 points. The oral interview will yield four scores ranging from 5 to 1, reflecting fluency. Prosodic features, accuracy ot the target grammatical objectives, and discourse approprarcness. 1oeignt test scores appropriately, you WI double each individual score and then add them together for a possible total score of 40. (Chapters 4 and 7 will deal more extensively with scoring and assessing oral production performance.)the writing sample has two scores: one for grammar/mechanics (including the correct use of so and because) and one for overall effectiveness of the message, each ranging from 5 to 1. Again, to achieve the correct weight for writing. You will double each score and add them, so the possible total is 20 points.

GRADING

Your first thought might be that assigning grades to student performance on thistest would be easy: just give an "A" for 90-100 percent, a "B° for 80-89 percent, and so on. Not so fast! Grading is such a thorny issue that all of Chapter 11 is devoted to the topic. How you aSsign letter grades to this test is a product ofthe country, culture, and context of this English classroom, institutional expectations (most of them unwritten), explicit and implicit definitions of grades that you have set forth, the relationship you have established with this class, and student expectations that have been engendered in previous tests andquizzes in this class.For the time being, then, we will set aside issues that deal with grading this test in particular, in favor of the comprehensive treatment of gracing.

GIVING FEEDBACK

A section on scoring and grading would not be complete without some consideration of the forms in which you will offer feedback to your students, feedback that you want to become beneficial washback. In the example test that we have been refer ring to here-which is not unusual in the universe of possible formats for periodic

EXERCISES

Note: (D ndividual work; (G) Group or pair work; (C) Whole-class discussion.]1. V)Consult the MLAT website address on page 44 and obtain as muchinformation as you can about the MLAT. Aptitude tests propose to predict one's performance ina language course. Review the rationale supporting such testing, and then summarize the controversy surrounding aptitude tests. What can you say about the validity and the ethics of aptitude testing2. (G) In pairs, each assigned to one type of test (aptitude, proficiency, place-ment, diagnostic, or achievement), create a list of broad specifications for the test type you have been assigned: What are the test criteria? What kinds of items should be used? How would you sample among a number ot possible objectives?3. (G) Look again at the discussion of objectives (page 49). In a small group,discuss the following Scenario: In the case that a teacher is faced with more objectives than are possible to sample in a test, draw up a set of guidelines for choosing which objectives to include on the test and which ones to exclude You might start with considering the issue of the relative importance of allNC ODjccuves in the context of the course in question. How does one ade sum pic objcctuves?4. (V)Figure 3.1 depicts various modes of elicitation and response. Are thereother modes of elicitation that could be included in such a chart? Justify your additions with an exampie o each.5. (G) Select a language class in your immediate environment for the followingproject: In small groups, design an achievement test for a segment of the course preferably a unit for which there is no current test or for which the present test is inadequate). Follow the guidelines in this chapter for facility P and item discrimination (1D) index for selected items. If there are structure of those items in a distractor analysis to determine if they have6. (G) Find an existing, recently uscd standardized multiple-choice test forwhich there is accessible data on student performance. Calculate the item no data for an existing test, select some items on the test and analyze the (a) any bad distractors, (6) any bad stems, or () more than one potentiallycorrect answer7. W) On page 63, nine diferent options are listed for giving feedback to students on assessments. Review the practicality of each and determine the extent to which practicality (principally, more time expended) is justifiably sacrificed in order to offer better washback to learners.FOR YOUR FURTHER READINGCarroll, John B. (1990). Cogniive abilities in foreign language aptitude: Then andnow. In Thomas s. Parry &Charles W. Stansfield (Eds.), Language aptitude recon sidered. Englewood Cliffs, NJ: Prentice Hall RegentsCarroll, the original developer of the MLAT, updates arguments for and against some of the original cognitive hypotheses underlying the MLAT. In the same volume, note articles by Oxford and by Ehrman contending that styles, strategies, and personality may be further factors in the construct of language aptitude.Brown.James Dean.

Source:

Brown, H. Douglas. 2003. Language Assessment Principles and Classroom Practices. San Francisco, California