High Quality Items for High Stake Tests

Delhi, 20 November, 2015

Consequences of using flawed test items

It has been shown that flawed items are up to 15 percentage points more difficult than standard items testing the same content, with the median passing rates about 3.5 percentage points more difficult than standard items.

High-stakes decisions are based on test performance on a sample of items.

The quality and effectiveness of test items is determining factor of the quality of test.

The world is awash in poorly written items.

- Trivial - Ambiguous - Irrelevant difficulty - Flaws benefitting testwise

candidates- Low cognitive level - Window dressing- Extremely difficult or easy items - Inflammatory, offensive or upsetting

What constitutes a high-quality test item?

The principles of effective writing are well delineated. These principles, based on a consensus of the educational measurement of textbook authors and empirical research studies can be considered the ‘evidence-based’ principles of effective item writing.

The ideal item assesses appropriate content accurately, consistently and fairly

Testing Tenants: Focus on Measurement Validity

ReliabilityFairness

- How well item measures the subject matter- Whether the items address what the successful

candidates need to know- What the incorrect answers are measuring

+ Are the items clearly written and unambiguous+ Does the item include content relevant to testing

Is the wording of the item free from sensitive issues Is the content of the item causing invalid differences between groups

Item writing is both an art and science

The creation of effective test items designed to measure content at an appropriate cognitive level is a nontrivial skill, requiring a great deal of professional time and effort from content experts and test developers.

The art of item writing:

Understanding the eligibility requirements to sit for the testSkillful application of item writing principles to the contentCreativity

The science of item writing:

Knowledge of item writing techniquesThe ability to follow item writing best practicesA sense of how item writing fits into the test development process.

Effective items are created, edited, reviewed, and analyzed

It is nearly impossible to estimate how an item will perform until it is actually used in a test. Item analysis provides information on what type of items tend to work well and what types tend not to work well in a particular test.

Characteristics and traits of effective item writers: - Content expertise - Writing skills

Effective item writers are trained not born: - Unique skill and must be learned and practiced - Extensive knowledge of evidence-based item writing principles

Item editing is a craft which improve the quality of item:- Improves the appearance, readability and fairness

- Item writer and professional editor work as a team

Item review by expert group to verify that- It assesses appropriate and important information and is accurate- Keyed answer is absolutely correct

Thank you.