waop presentation 2016 a basket case
TRANSCRIPT
![Page 1: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/1.jpg)
eelloo.nl
A basket caseUsing baskets and verification in unproctored testing
Jozien SilvesterLiesbeth van LeeuwenJelle Geertsma
WAOP Presentation 25 november 2016
![Page 2: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/2.jpg)
eelloo.nl
Unproctored testing
• Stakes are high in selection
• Possibility of cheating
• Proctoring not always possible or efficient
• Benefits for candidates of unproctored testing: time and place of choice
![Page 3: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/3.jpg)
eelloo.nl
Unproctored testing
• Validity proctored /unproctored equal if:
• Informing candidates about optimal test conditions
• Taking measures to prevent fraud
• Items that measure reasoning (fluid)
• Time limit
• Basket method
![Page 4: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/4.jpg)
eelloo.nl
Basket method
• Large item bank
• Items of equal difficulty are placed in a basket
• Item is randomly picked from basket
• Same number of baskets for every candidate
• Four subtests: Figure series, Analogies, Diagrams, Number series
![Page 5: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/5.jpg)
eelloo.nl
Benefits of basket method
• Easy calculation
• Easy monitoring, maintaining, extension
• Easy retesting (system remembers which items were already administered)
• Prevention of item leakage
![Page 6: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/6.jpg)
eelloo.nl
Validity
• High correlation with proctored extended test (.75)
• Test-retest correlation (.77)
• About 8% scores lower than expected, possible cheaters
![Page 7: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/7.jpg)
eelloo.nl
Basket monitoring
![Page 8: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/8.jpg)
eelloo.nl
Basket monitoring
• Some items with equal difficulty are more equal than others
Basket Item Difficulty (n) Difference Average
1017 Hcyf106b 0.66 (287) 0.10 0.68
Hcyf106c 0.65 (266)
HCYF01b 0.74 (294)
1018 HCYF112 0.62 (409) 0.02 0.63
HCYF112b 0.64 (438)
![Page 9: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/9.jpg)
eelloo.nl
Difference in p-value
• Even if items are carefully balanced, there will still bedifferences, although they may be small
• For most candidates this will average
• Some will have all the ‘easy’ or all the ‘difficult’ items
• Effect on reliability and score
• Comparison of correction methods on two measures
![Page 10: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/10.jpg)
eelloo.nl
Score calculation improvement - reliabilities (split half)
Education level N CTT IRT1pl IRT2pl Elo Glicko
MBO (Vocational) 743 0.85 0.80 0.81 0.79 0.86
HBO (Bachelor) 1397 0.82 0.82 0.83 0.79 0.85
WO (Master) 706 0.80 0.76 0.77 0.74 0.81
• Glicko has best performance
• CTT has good results as well
![Page 11: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/11.jpg)
eelloo.nl
Effects Glicko on z-score
Item difficulty group
regular scoreGlicko score
z-sc
ore
![Page 12: Waop presentation 2016 a basket case](https://reader031.vdocuments.mx/reader031/viewer/2022030402/588937b21a28ab22578b5ef1/html5/thumbnails/12.jpg)
eelloo.nl
Concluding remarks
• Valid unproctored test possible with basket method
• Glicko shows slight improvement in score calculation, but CTT also works well