leveraging tech enhanced items without sacrificing psychometrics
TRANSCRIPT
#atpconf
The Best of Both Worlds: Leveraging TEIs Without Sacrificing
PsychometricsNathan Thompson, Karen Peters, David Saben, Terry Ausman
Assessment SystemsCharles County Public Schools
2#atpconf
Tech-Enhanced Items (TEIs)
■ Technology has enabled a brave new world in item development
■ Lots of effort to use tech
■ But what is the real goal?
12/2/2014
3#atpconf
The future of open-response items?
12/2/2014
4#atpconf
Need to adapt way of thinking
12/2/2014
5#atpconf
The Landscape
■ Education PARCC SBAC ESSA PISA
■ Pre-Employment Fidelity
■ Certification NCCA and ANSI
12/2/2014
6#atpconf
The Scenario
■ Pre, Post, and Formative■Many courses & tests■ CBT and Paper■ Students scored with IRT and scaled■ Need to report scores immediately or
quickly
12/2/2014
7#atpconf
The Scenario
■ All work is done in house Item development Form assembly Delivery ECR marking Scoring and reporting
12/2/2014
8#atpconf
The Scenario
■Maryland is in PARCC■ Several years ago, we faced a need to
stay current■ TEIs were one of the big issues■Other major needs:
Essay Scoring module Role-based score reporting module Formative bank
12/2/2014
9#atpconf12/2/2014
10#atpconf12/2/2014
11#atpconf12/2/2014
12#atpconf12/2/2014
13#atpconf12/2/2014
14#atpconf
The Scenario■ High level need:
A system that allows a single school district to develop a ton of tests at a professional level like a testing company
■ Specific needs: PARCC TEIs scored
with IRT in real time PARCC ECRs scored
with IRT as quickly as possible
(and more!)
12/2/2014
15#atpconf
The Solution
■ PARCC-style TEIs■ First, categorize functionality
Drag and drop image- Number line- Bivariate plane- Labeling map/figure/diagram
Drag and drop text to lists- Single list- Multi list
Drag and drop to text blanks- Ranking/Sequence- Simple equations
Equation editor12/2/2014
16#atpconf
The Solution
■ Constructed response Brief CR- Real time scoring- Dichotomous typically
Extended CR- Post-window scoring- Polytomous typically
12/2/2014
17#atpconf
The Solution
■ IRT What models are needed?- Dichotomous models- Generalized partial credit model
Where will they be defined for usage? How are they calibrated?
12/2/2014
18#atpconf12/2/2014
19#atpconf
The Solution
■ IRT What functionality is needed?- System knows which model is relevant for item types- Estimate item parameters from data- Store item parameters in item bank- Use item parameters for test assembly- Use item parameters for scoring
12/2/2014
20#atpconf
The Solution
■Design and implementation Build everything above!
12/2/2014
21#atpconf12/2/2014
22#atpconf12/2/2014
23#atpconf12/2/2014
24#atpconf12/2/2014
25#atpconf12/2/2014
26#atpconf
The Solution
■ Re-train item writers: Paper No, we can’t just
deliver all tests on paper and scan in images!
12/2/2014
27#atpconf
The Solution
■ Re-train item writers: Rethink Rethink how you construct items Before: Define a linear equation that ___- Tell student to write it down
Now: Define a linear equation that ___- Drag and drop symbols to boxes- Use an interactive equation editor- Can we do even better?
12/2/2014
28#atpconf
The Solution
■ Re-train item writers: Scoring In previous example, how will you define scoring
for either tech approach?- Each blank worth 1 point?- If equation editor, exact or evaluated?
Integer scores only!
12/2/2014
29#atpconf
The Solution
■ Items that must be hand-scored Rubric definitions (must be reusable) Rubrics assigned to items Item marking- Dashboards- Teacher screens- Admin screens
Obtaining results
12/2/2014
30#atpconf
The Recap
■What was our goal?
■What was our need?
■What was the solution?
■ How did it turn out?
12/2/2014
31#atpconf
The Recap
■ There are lots of item types possible! Choose wisely.
■ Keep the goal in mind■ Don’t forget the item
writers
12/2/2014
32#atpconf
The Recap
What challenges have you faced?
12/2/2014
