CLEAR 2008 Annual Conference
Anchorage, Alaska
“Problems and Priorities in Pretesting Pondered”
Beth Noeller
Grady Barnhill
Carol O’Byrne
CLEAR 2008 Annual ConferenceAnchorage, Alaska
CLEAR 2008 Annual ConferenceAnchorage, Alaska
All Items are not Created Equal
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Purposes of Pretesting
Gather performance information
Add new content
CLEAR 2008 Annual ConferenceAnchorage, Alaska
What Information do you Gather?
• Performance data♦ Classical and IRT
♦ Difficulty
♦ Discrimination
• Cognitive Level Data
• Content
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Difficulty
• How hard is the item?
• Usually expressed as a %.
• What percent of the candidates get the item correct?
• IRT b-value
CLEAR 2008 Annual ConferenceAnchorage, Alaska
What is a “good” difficulty?
• What is your cut score?
• The best difficulty is right around the cut score.
• Too easy and too hard provide little information.
• Between 40 & 90 for many testing programs
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Discrimination
• How well an item differentiates between those who perform well on the test and those who get the item correct.
• Correlation• Values from -1.0 to +1.0• IRT a-value
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Positive Discrimination Values
• Two people walking together ‘in step’
• Get an item correct and receive a high score on the examination
• Perfect positive relationship +1.0
Item correctness
exam score
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Negative Discrimination Values
• Two people walking in opposite directions
• Get an item correct and receive a low score on the examination
• Perfect negative relationship -1.0
Item correctness
exam score
CLEAR 2008 Annual ConferenceAnchorage, Alaska
What is a “good” discrimination?
• A positive relationship
• The higher the better
• Diversity of the candidate population
• Greater than 0.15
exam score
Item correctness
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Combining Difficulty and Disc.
• Each quality by itself is linear
• Together they can be depicted on a two-dimensional graph
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Nursing
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Health Related 4 yr +
-0.35
-0.25
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0.55
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Health Related2 yr
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Heath Related 4 yr
-0.45
-0.35
-0.25
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0.55
0.65
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Health Related6 yrs +
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Health Related6 yr +
-0.35
-0.25
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
New Medical
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0.55
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
New Medical
-0.1
0
0.1
0.2
0.3
0.4
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Mature Medical
-0.25
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Health Related
-0.25
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0.55
0.65
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Non-Health
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Non-HealthMath
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Professionalnew @ pretesting
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
CLEAR 2008 Annual ConferenceAnchorage, Alaska
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Professional
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
0 10 20 30 40 50 60 70 80 90 100
Difficulty
Dis
crim
inat
ion
CLEAR 2008 Annual ConferenceAnchorage, Alaska
No. Pretest Items
Survival %
% neg. disc
% low disc
% easy % hard
2900 44% 10% 33% 2% 11%1000 40% 8% 36% 12% 3%760 58% 6% 24% 9% 4%597 42% 15% 30% 10% 4%922 46% 13% 36% 8% 3%
1325 31% 17% 41% 8% 3%294 55% 5% 32% 15% 3%197 46% 2% 35% 7% 11%800 41% 15% 35% 4% 5%622 35% 13% 44% 5% 4%810 63% 0% 2% 35% 0%430 68% 0% 1% 0% 31%240 56% 3% 28% 5% 8%
1147 58% 5% 22% 6% 9%100 6% 13% 54% 14% 13%46 28% 22% 39% 7% 4%
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Summary
No. Pretest Items
Survival %
% neg. disc
% low disc
% easy % hard
12,190 45% 9% 31% 9% 7%
Range 6-68% 0-22% 1-54% 0-35% 0-31%
CLEAR 2008 Annual ConferenceAnchorage, Alaska
How do you use pretesting to add new content?
• Get the pulse of your item bank
• Inventory of acceptable scored items
• Inventory of unused items
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Scored Items and Test Specs
12%
40%
10%
17%
21%
18%
35%
8%
20%
19%
I
II
III
IV
V
Inner Circle = Content Specifications
Outer Circle = Used Pool
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Pretest Needs
12%
40%
10%
17%
21%
6%
46%
12%
13%
23%
I
II
III
IV
V
Inner Circle = Content Specifications
CLEAR 2008 Annual ConferenceAnchorage, Alaska
Beth’s Contact Information
Beth Noeller, PhD
319-339-3095
