a procedure for dimensionality analyses of response data from various test designs jinming zhang...
TRANSCRIPT
![Page 1: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/1.jpg)
A procedure for dimensionality analyses of response data from
various test designs
Jinming Zhang
William Stout
![Page 2: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/2.jpg)
Introduction
Dimension Dimensional structure of the test (e.g.,
algebra and geometry) statistical dimensional structure of response
data Incorporate both Judgments about test
content and evidence from statistical analyses
Missing data: CAT & multistage testing Missing item pair measurement
![Page 3: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/3.jpg)
![Page 4: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/4.jpg)
![Page 5: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/5.jpg)
DETECT index
![Page 6: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/6.jpg)
Modified DETECT index
![Page 7: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/7.jpg)
Example 3 Stage 1 booklet: 2 dim & Stage 2 booklet 1: 2
dim, Stage 2 booklet 2: 1 dim (additional dim), Stage 2 booklet 3: 1 dim (additional dim).
Bridge item (in Stage 1) E(1,2) and E(2,3), so E(1,3)
First-stage booklet should measure all of the constructs/contents the whole
test aim to measure, though it is unknown. classify examinees into different proficiency levels
![Page 8: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/8.jpg)
Item 1 and 2 are measuring the same dimension
1. If E(1,2), E(2,3), and E(1,3) D(P*) is maximized in P* partition if item 1 and
2 are in the same cluster, other than in other P
2. If only E(1,2) and E(2,3), and if item 2 is a bridge item
D(P*) is maximized in P* partition if item 1 and 2 are in the same cluster when item 2 is in the same cluster or not, other than in other P
![Page 9: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/9.jpg)
![Page 10: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/10.jpg)
Whether a test has an approximate simple structure or not
![Page 11: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/11.jpg)
Discordance Resulted from
Not a approximate simple structure Inaccuracy of conditional covariance
estimation Given a partition of items,
![Page 12: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/12.jpg)
![Page 13: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/13.jpg)
What made Dd(P*) and PropD(P*) large Unidimensionality Violation of approximate simple
structure Inaccuracy of conditional covariance
estimation
![Page 14: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/14.jpg)
polyDETECT
Obtain the composite theta score: unidimensinoal approximation & simple structure approach.
Use percentiles of composite theta scores as cut-off points in forming AHGs. Between 25 to 100 in each group
Cross-validation:
![Page 15: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/15.jpg)
polyDETECT
Evidence of multidimensionality
![Page 16: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/16.jpg)
Condition 3 is hold or not? Each sizable cluster contains at least one
stage-1 item. Exist at least one sizable cluster that does not
contain any Stage-1 items, and all items in such cluster belong to the same booklet.
Exist at least one sizable cluster that does not contain any Stage-1 items, and all items in such cluster belong to at least two booklet.
Exist at least two sizable cluster that does not contain any Stage-1 items, items in each such cluster come from the same booklet but different clusters belong to different booklets.
![Page 17: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/17.jpg)
Dealing with CAT data
Estimates of item parameters were obtained before conducting CAT. Why do dimensionality assessment on CAT data?
Sparse data set of CAT 100,000 responses are required at
least Item selection, item exposure
control, content balance to satisfy the condition 3
![Page 18: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/18.jpg)
Simulation study M2PL Each booklet has 30 items Dimension: 1, 2, 3 Number of examinees: 750, 1500, 3000, 4000 Theta: MVN(0,sigma), correlation = 0.8 Cut-off points for low-, moderate-, and high-
scoring group: <10, 11~18, >18 About 37.82% unestimable item pairs Cross-validation Replication: 100 Composite theta: use unidimensional IRT
model
![Page 19: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/19.jpg)
Results
![Page 20: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/20.jpg)
Real data analyses
Missing values are large (55%~71%)
![Page 21: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/21.jpg)
Real data analyses
Weak dimensionality (M value) High PropD(P*) indicates a large amount of
spurious information to form partition P* Confirmatory analyses: D3 >D2
But DETECT tends to underestimate, so two-cluster partition solution may be preferred.
![Page 22: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/22.jpg)
Real data analyses
High correlation indicates a weak degree of multidimensionality
![Page 23: A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d015503460f949d3c89/html5/thumbnails/23.jpg)
Concluding remarks
Moderate violation of approximate simple structure is still hold