Module 6

Test Construction &Evaluation

Lesson’s focus

Stages in Test Construction

Tasks in Test

Test Evaluation

Questions to ask yourself in designing a Test

What objectives will (should) I be testing?

What types of items will be included in the test?

How long will the test be in terms of time and

number of items?

How much will each objective be worth in terms

of weighting and number of items?

Writing a test specification

Stages in Test Construction Order the following stages in the test construction.

- Decide on the marks- Make a key- Decide what kind of test it is- Find appropriate text- Pilot the test- Write a list of what the test covers- Weigh the sections in the order of importance and timing- Write the marking scheme- Analyse the test and decide what can be kept/rejected- Review and revise the test and key- Write the questions- Think about length, layout and format- Write the instructions and examples

Stages in Test ConstructionSuggested order

- Decide what kind of test it is- Write a list of what the test covers- Think about length, layout and format- Find appropriate text- Weigh the sections in the order of importance and timing- Write the questions- Write the instructions and examples- Decide on the marks- Make a key- Pilot the test- Write the marking scheme- Review and revise the test and key- Analyse the test and decide what can be kept/rejected

Which stages can be repeated several times? Why?

Question Types vs Levels of Performance

Knowledge

Comprehension

Application Analysis

Synthesis

Evaluation

Multiple Choice (MC)

True/False (TF)

Matching

Completion

Short Answer

…

MC

Short Answer

Problems

Essay

…

MC

Short Answer

Essay

…

Can you add more to the columns?

Multiple Choice Items

Advantages: Can measure the higher level mental processes

(application, analysis, synthesis and evaluation) Can cover a wide range of content Efficiently and reliably scored Different response alternatives can provide diagnostic

feedback (item analysis)

Disadvantages Difficult to construct plausible alternative responses

Types of Multiple Choice Items

Four Basic Types Question Type Incomplete Statement Type Right Answer Type Best Answer Type

1. The capital city of Canada is1. The capital city of Canada is a. Vancouvera. Vancouver b. Montrealb. Montreal c. Torontoc. Toronto*d. Ottawa*d. Ottawa

Terminology: Multiple Choice

1. Stem: presents the problem

2. Keyed Response: correct or best answer

3. Distracters: appear to be reasonable answers to the

examinee who does not know the content

4. Options: include the distracters and the keyed response.

44

11

22

33

Guidelines for Writing MC

Place most of the subject matter in the Stem ensures full statement of problem

Avoid Negatively phrased Stems students may miss the qualifier

use only when learning outcome requires this type of differentiation

Ensure similarity among alternatives with regard to: grammatical structure length mode of expression

Grammatical errors provide unintentional clues to the answer When in doubt, students will select the longest alternative as

the correct answer

Example: Length of Alternatives

Neurotics are more likely than psychotics to

a. be dangerous to societyb. have delusional symptomsc. be dangerous to themselves*d. have insight into their own inappropriate

behavior but nevertheless feel rather helpless in terms of dealing with

their difficulties

Guidelines for Writing MC Randomly distribute answers across the

alternative positions inexperienced test writers emphasize “b” and “c” alternatives

do NOT use an interpretable order of keyed responses Response alternatives should not overlap,

logically or numerically ordered Avoid “All of the above” and “None of the above” Avoid Negatively phrased Stems

students may miss the qualifier

use only when learning outcome requires this type of

differentiation

What is the cost of an item that normally sells for 9.99 USD that is discounted 25%?

Order Options Numerically

WrongA. 5.00B. 7.50C. 2.50D. 6.66

RightA. 2.50 B. 5.00C. 6.66D.7.50

What are the three important concerns in achievement testing? V=validity R=reliability E=efficiency O=objectivity

Order Options Logically

WrongA. V, R, E B. O, V, RC. R, O, E D. E, O, V

RightA. V, R, O B. V, R, E C. V, E, O D.R, E, O

True-False Test Items

Best suited for testing 3 kinds of info.: Knowledge level learning Understanding of misconceptions When there are two logical responses

Advantages: Sample a large amount of learning per unit of student

testing time Disadvantages:

Tends to be very easy 50-50 chance of guessing Tends to be low in reliability

Tips for constructing True-False Items

o Provide a “T” and “F” beside each statement and ask

students to circle correct answer (students writing

illegible letters).

o Include background, qualifications, and context

"According to...., ...."

o Paraphrase, and do not directly quote

o Every part of a true sentence must be "true"

Tips for constructing True-False Items

- Avoid Unfamiliar vocabulary and concepts

Double negatives*

Specific determiners (always, never, only, etc.) =

overgeneralization*

long or complex sentences

Two ideas in each statement*

Exact quantitative (two, three, four) language is better

than qualitative (some, few, many)

A pattern of answers

True / False

negative words

Poor: It was not unheard of for Henry VIII to close

monasteries in England.T F

Better: Henry VIII closed some monasteries in England.

(T)

“never”, “none”, “always”, and “all”.

Testwise students will recognize that there are few

absolutes.

True / False

Two ideas in one statement Poor: Porpoises are able to communicate because

they are mammals. T F

Better: Porpoises are mammals. T F

Porpoises are able to communicate. T F

Test Evaluation

List of criteria for a good test:- Test purpose- Test format- Test Content- Test Validity- Test Reliability- Test Practicality- Test washback

Test Evaluation Below is the comment note of an experienced

teacher on a reading test for grade 10 students.

Read and tell what criteria have been broken:

C1: Questions 5-10 tend to test students’ grammar

rather than reading comprehension.

C2: Questions 21-23 try to test the specialized

knowledge of geography rather than language

perception and use.

C3: The checkbox for true/false questions is rather

confusing since it is not in parallel with the

statements.

Test Evaluation

C4: The test seems to be a proficiency test

than an achievement test.

C5: Some open questions seek for a very

long answer or many possible answers,

then students find it hard to write them all

and teachers find it hard to score.

C6:The test gives students a feeling that

they can get high marks if they focus on

learning grammar and vocabulary rather

than reading skills.

Summary

Stages in Test Construction: Purpose Plan Items Pilot Analyse Revise (Pilot, Analysis and Revision: repeated some time)

Tasks in Test: Multiple Choice, Gap-filling, True/False, Open question, etc.

Test Evaluation: purpose, format, content, validity, reliability, practicality, and washback