NRTs and CRTs

Group members: Camila, Ariel, Annie, William

NRTs—Norm-referenced Item Analysis

Item facility Item discrimination

Why do we do item analysis?

Assemble a large number of items of the type you want on the test

make sure the items are well written and clear

Pilot the items Analyze the results of the pilot testing Select the most effective items/get rid

of the ineffective items or revise the weak ones

The basic purpose of NRTs

To spread students out along a general continuum of language abilities

making aptitude, proficiency, or placement decisions

Two item statistics used in IA of NRTS

Item facility(IF): the proportion of students who answered a particular item correctly.

45/50=.90---90% of the students answered the item correctly—the item is very easy

Item discrimination(ID):Caculate IF for the upper group and the lower group using AVERAGE (C2:C6) and AVERAGE (C15:C19) ID=IFu-IFl

ex: IFu-IFl=.20

Item Discrimination (ID)

can be calculated by first figuring out who the upper and lower students are on the test

using their total scores to sort them from the highest score to the lowest

Equal numbers of students in three groups—high/middle/low

ID=Ifupper - Iflower

Ideal items in an NRT

Should have an average IF of .50 50% of the students answered correctly

and 50% of them answered incorrectly In reality, items rarely have an IF of

exactly .50 Those that fall in a range between .30

and .70 are usually considered acceptable for NRT purpose

Items within .30 to .70 range

The items among them that have the highest IDs should be further selected for inclusion in the revised test.

This process would help the test designer to keep only those items that are well centered and discriminate well between the high and the low scoring students

Conclusion for NRT

IF and ID are only appropriate for developing and analyzing norm-referenced tests

Used at the institutional level overall English language proficiency tests or placement

tests

Not appropriate for developing and analyzing classroom oriented criterion-referenced tests like the diagnostic, progress, and achievement tests

CRTs-Criterion-referenced Item Analysis

Purpose of CRTs—to measure the amount (or percent) of material in a course or program of study that students know

Usually for purposes of making diagnostic, progress, or achievement

Two item statistics used in IA of CRTs

1.The difference index(DI): the item facility on the particular item for the posttest minus the item facility for that same item on the pretest

pretest:10/50=.20; posttest:45/50=.90 DI=.90-.20=.70 DI tells how much the students are

improving between the pretest and posttest on each item

The higher the value of the DI, the better. A value of 1.00 is a perfect difference

index

Two item statistics used in IA of CRTs

2.The B-index: the item facility on the particular item for the students who passed the test minus the item facility for the students who failed

The B-index show how well each item is contributing to the pass/fail decisions that are often made with CRTs

B-index=IFpass - IFfail=14/14-0/6=1.0-.00=1

P.21

The B-index tell us how well each item is contributing to the pass/fail decision on this test at the cut-point.

Like ID and DI statistics, the higher the B-index, the better.

A perfect B-index would be 1.00.

Conclusion-CRTs

When should these indices be used? To analyze the items on a CRTs for

purposes of revising the test The items with the highest values

should generally be kept in both cases, DI and BI

Making these decisions not as simple as it is for NRT development, because a CRT item may not be performing well in terms of these statistics for many reasons.

Reasons that a CRT item may not be performing well

The item is written/ working poorly The objective the item is testing is vague Ss are not ready to learn this particular

objective One/ all of the teachers did not teach this

particular objective or teaching it poorly The materials are confusing with regard to

this particular objective The statistics cannot tell you exactly what

is wrong even though they can point you to places in your curriculum where something is not working well

Some common-sense analysis of the entire situation needs to be done

To revise the CRT or other aspects of your curriculum such as the objectives, the materials, the teaching, etc.,

What the statistics help you?

Help you figure out where to focus your energies

The DI will tell you how well each item fits the objectives of your curriculum

The BI will tell you how each item is contributing to the pass/fail decision that you must make at whatever cut-point you are using.

2 Definitions for “Criterion” in CRT

1.Refers to the material being taught in the course

--CRT would assess the particular learning points of a particular course or program

This definition fits very well with the difference index, which indicates how well each item fits objectives of the curriculum

2 Definitions for “Criterion” in CRT

2. The standard of performance (or cut-point for decision making) that is expected for passing the test/course

--CRTs –used to assess whether students pass or fail at a certain criterion level (or cut-point)

Fits very well with the B-index, which indicates how well each item is contributing to the pass/fail decision that you must make at whatever cut-point your are making

What should you focus?

If you are interested in the degree to which your items are reflecting the material in your course….DI

If you are interested in the degree to which your items are helping you make decisions at a certain cut-point… BI

If you are interested in both statistics… DI and BI

What should they not be used?

Not used to analyze the effectiveness of norm-referenced items

What is the Ultimate Goal?

To produce a curriculum and CRTs that match each other such that you get high difference indexes and high B-indexes.