Testing 05

Test Methods

Considerations in Test Methods • Like traits tested, test methods also affect test

performance.• Test methods: features of the test context, the

controlled versions of the contextual features that determine the nature of the language performance that is expected for a given test.

• The closer the correspondence between the characteristics of the test method and the essential features of language use contexts, the more authentic the test task will be for test takers.

Some Common Test Methods• multiple choice• fill-in• cloze• translation—sentence• translation—passage• composition• dictation• short question answering• oral interview

Factors Affecting the Test Performance

• Test methods: Bachman (1981) found that scores from self-ratings loaded consistently more highly on method factors than on specific traits or ability factors.

Factors Affecting the Test Performance

• Individual attributes: test takers’ cognitive and affective characteristics, real world knowledge, age, sex, native language, educational and socio-economic background.

• Prior experience with the test, or test-wiseness: pacing, reading questions before the passages, ruling out alternatives. Such strategies are effective in reducing the unfamiliarity of the test. The provision of careful instructions with examples serves the same purpose.

Framework of Test Method Facets

• Testing environment

• Test rubric

• Input

• Expected response

• Relationship between input and response

Testing Environment

• Familiarity of the place and equipment

• Personnel

• Time of testing: morning or afternoon

• Physical conditions

Test Rubric

• Test organization• Salience of parts: test takers may adopt differing

response strategies.• Sequence of parts: from the easiest to the most

difficult or of the same difficult level• Relative importance of parts: weight, percentage

of the total score• Time allocation: the speed of test taking can be a

function.

Test Rubric

• Instructions• Language: native or foreign• Channel: visual or aural• Specification of procedures and tasks: test takers c

an perform better whent hey clearly understand the task that is required of them.

• Explicitness of criteria for correctness: test taker’ knowledge of criteria can affect the test performance directly.

Input & Expected Response

• Format

• Nature of language

Format • Channel & mode of presentation: visual, aural, ora

l• Form: language, nonlanguage• Vehicle: live human input (BEC oral exam) or can

ned human input (TSE)• Language: target vs. native• Identification of problem: underlined error identifi

cation vs. error correction of a line• Degree of speededness: Difficulty in speededness

may lead to guessing, or compensatory test taking strategies.

Nature of Language • Length: the longer the sample, the greater the pote

ntial effects of the other characteristics• Prepositional content• Vocabulary: frequency, specialization• Degree of contextualization: language use in a con

text rich with familiar information is a context-embedded discourse. (Context-reduced) the more context-embedded the input is, the more likely the test taker will be able to respond to its prepositional content.

Nature of Language

• Distribution of new information: highly compact input or highly diffuse input are both difficult to process.

• Type of information: concrete/abstract, positive/negative, factual/counterfactual

Nature of Language

• Topic: interesting, relevant, neutral. Test bias should be avoided.

• Genre: formal characteristics that are traditionally recognized. Differences in genres affect the interpretation of discourse. Particular types of language tests constitute genres.

Organizational Characteristics

• Grammar

• Cohesion

• Rhetorical

• The longer the language sample, the greater the need to incorporate organizational characteristics to make it interpretable.

Pragmatic Characteristics

• Illocutionary force: functionality of a given test task is the authenticity of the test, but some test task does not need the interpretation of illocutionary force for correct completion.

Sociolinguistic Characteristics

• Dialect or variety: standard language, American English, British English

• Register: test results may be misleading if the register of the test language is not appropriate to the register of the target language use context.

Restrictions on Expected Response

• The language used in language tests is sometimes characterized as “non-natural”, “non-normal”, “artificial”. It is restricted in these five aspects.

• Channel: noise in a real environment• Format: in some tests the format is highly

restricted, as in selection or identification response, while in others the format may be fairly unrestricted, as in composition test.

Restrictions on Expected Response

• Organizational characteristics• Grammar: in multiple choice test, test takers

deal with the meanings of a specific pair of words without the opportunity to express the equivalent meaning in their own words.

• Organization of discourse: in role play, test takers have to pretend to be a person he or she is actually not.

Relationship Between Input and Response

• Reciprocal input and response: with feedback as in oral interview

• Nonreciprocal input and response: without feedback as in most of the test

• Adaptive input and response: the input is influenced by the response, but without the feedback as in new TOFLE

Applications of Framework to Language Testing

• Description of language tests: describe or compare two tests or a test and a language program, e.G. TOEFL and CPE in terms of number of words and clauses, complex sentences, illocutionary acts, abstract, negative.

• Language test design: in test design, we often use tables of specifications about the components of ability to be tested, but the tables often fail to specify the facets of the test tasks.

Applications of Framework to Language Testing

• Validation of language tests: the validation must be based on a detailed description of both the abilities to be measured and the facets of the test methods.

• Formulation of hypotheses for language testing research: examination of the effects of specific facets, e.g. research of cloze test—deletion ratio, type of deletion, type of response, scoring criterion, organization of passage, passage content and discourse organization.