Virginia’s Perspective on Scoring for Technology … on Scoring for Technology Enhanced Items ... (SOL) assessments TEIs ... each response pattern be worth? 4)
39
Virginia’s Perspective on Scoring for Technology Enhanced Items CCSSO National Conference on Student Assessment 22 June 2016 Image by Mrs. E Teaches Math www.mrseteachesmath.blogspot.com
Virginia’s Perspective on Scoring for Technology Enhanced Items CCSSO National Conference on Student Assessment
22 June 2016 1
Image by Mrs. E Teaches Math www.mrseteachesmath.blogspot.com
Online Testing in Virginia
0
500000
1000000
1500000
2000000
2500000
3000000
Fall 2001
Spr 2003
Spr 2005
Spr 2007
Spr 2009
Spr 2011
Spr 2013
Spr 2015
1700 8500 76000 265000
579000
924050
1333150 1418250
1652680
1908970
2216690
2489170 2656010 2661406
2427599 *
Presenter
Presentation Notes
*Note that Spring 2015 testing volumes are somewhat lower because the General Assembly eliminated 5 tests that had been administered online.
Background
Beginning in the Spring of 2011, Virginia began field testing Technology Enhanced Items (TEIs) as part of the Standards of Learning (SOL) assessments TEIs now make up 15-20% of the items on all of the
SOL tests Currently students are awarded either full credit (1
point) or no credit (0 points) for both multiple choice items and TEIs Recent interest from the SOL Innovation
Committee has encouraged consideration of offering partial credit for TEIs
3
Background
The Standards of Learning Innovation Committee was created by HB 930 in the 2014 legislative session. Purpose of the Committee is to provide the Board
of Education and General Assembly with suggestions on changes to the SOL assessments, authentic individual student growth measures, alignment between the Standards of Learning and assessment, and ideas on innovative teaching in the classroom.
4
Background
SOL Committee Recommendation (2015):
“…all assessments should be scored fairly and accurately, with partial credit being awarded for assessment items that require multiple responses.”
5
Partial Credit Scoring and TEIs
Seems like a logical recommendation
6
Partial Credit Scoring and TEIs
However…implementation is more complex than it might seem with lots of decisions to be made about how and when to award partial credit
7
Audience Participation Section
Example 1: Drag and Drop with 2 Bays
9
Presenter
Presentation Notes
Our first example is a drag and drop item where a student will drag one of the six options in the lower box to each of the two bays (response areas) in the question. Take a moment to read and think about your answer to this item. It’s a 3rd grade math item so shouldn’t be too difficult.
Example 1: Drag and Drop with 2 Bays Full Credit
10
Presenter
Presentation Notes
Here is the “correct” answer which would earn you full credit? How many people answered this one correctly and would have received full credit? Do you think partial credit should be awarded for this item? If so, what student responses would qualify for partial credit?
Example 1: Drag and Drop with 2 Bays Partial Credit?
11
Presenter
Presentation Notes
Here’s an example which might be awarded partial credit. The first response (ten thousands) is correct, but the second responses (ones) is not correct. Students who had this response might be awarded partial credit. Similarly students who answered the second response correctly, but not the first might also be awarded partial credit.
Example 2: Drag and Drop with 2 Bays
12
Presenter
Presentation Notes
Here’s another item which is also an example of a drag and drop item. Like the first item it has two response areas (bays). Take a moment to read and think of your answer.
Example 2: Drag and Drop with 2 Bays Full Credit
13
Presenter
Presentation Notes
Here is the correct response that would earn full credit? Do you think partial credit should be awarded for this item? If so, what student responses would qualify for partial credit? In this case, the two response areas are related to each other and you can’t get a partially correct answer. So it would not make sense to award partial credit for this item even though it is a TEI and requires multiple responses.
Example 3: Drag and Drop with 8 Bays
14
Presenter
Presentation Notes
This example shows a drag and drop item with 8 bays. Take a moment to read and think of your response.
Example 3: Drag and Drop with 8 Bays Full Credit
15
Presenter
Presentation Notes
Here is the correct answer that would earn full credit. Are there responses that should earn students partial credit?
Example 3: Drag and Drop with 8 Bays Partial Credit?
16
Presenter
Presentation Notes
Here the student has correctly labeled 5 of the 8 response bays. The responses labeled Saturn, Uranus, and Earth are incorrect. Should this response be awarded partial credit? How much partial credit should it be awarded? Should students get 5 points for this item because they got 5 of the 8 bays correctly labeled? Does that give too much weight to this item? Should students get half credit for this item because they got more than half correctly labeled?
Example 4: Drag and Drop with Order
17
Presenter
Presentation Notes
Here is another drag and drop item. In this example, there are 5 bays and students must place the draggers in the bays in the correct order. Take a moment to read the question and think about your answer.
Example 4: Drag and Drop with Order Full Credit
18
Presenter
Presentation Notes
This is the correct response and would be awarded full credit. Are there ways you think students could respond which should be awarded partial credit?
Example 4: Drag and Drop with Order Partial Credit?
19
Presenter
Presentation Notes
In this response the student has reversed “route” and “rose” in their response. But they have correctly ordered the other 3 responses. Should this response receive partial credit?
Example 4: Drag and Drop with Order Partial Credit?
20
Presenter
Presentation Notes
In this response the student has 4 of the responses in the correct order, but starts with the wrong response. Row should be last not first. Should this response receive partial credit?
Example 4: Drag and Drop with Order Partial Credit?
21
Presenter
Presentation Notes
This response has the answers in reverse alphabetical order. Should this response receive partial credit?
Example 5: Hot Spot
22
Presenter
Presentation Notes
Here is a hot spot item where students are asked to select the chapters which answer the question. Take a moment to read the question and think about your answer.
Example 5: Hot Spot Full Credit
23
Presenter
Presentation Notes
Here is the correct answer which would be awarded full credit. Are there ways that you think students could respond to this item which should be awarded partial credit?
Example 5: Hot Spot Partial Credit?
24
Presenter
Presentation Notes
Here the student has selected one correct answer choice and one incorrect answer choice. Should this response be awarded partial credit?
Example 5: Hot Spot Partial Credit?
25
Presenter
Presentation Notes
Here the student has selected both correct answers, but has also selected one incorrect answer choice. Should this response be awarded partial credit?
Example 5: Hot Spot Partial Credit?
26
Presenter
Presentation Notes
Here the student has selected both correct answer choices, but has also selected all the incorrect answer choices. Should this response be awarded partial credit? Note that new item development for VA SOL will specify the number of correct answers rather than saying “select all” so this will limit the number of possible response patterns.
Example 6: Bar Chart
27
Presenter
Presentation Notes
This is an example of a bar chart item where students are asked to graph the data in the table. Take a moment to read the question and think about how you would answer.
Example 6: Bar Chart Full Credit
28
Presenter
Presentation Notes
This is the correct answer and would receive full credit. Are there responses that you think should be awarded partial credit?
Example 6: Bar Chart Partial Credit?
29
Presenter
Presentation Notes
In this response the student has correctly positioned 3 of the 4 bars. The bar for Kristen’s data is incorrectly positioned at 7 rather than 8. Should this response receive partial credit? If so, how much? Should the student receive 3 points for this item?
Example 6: Bar Chart Partial Credit?
30
Presenter
Presentation Notes
In this response the student has correctly positioned 2 of the 4 bars. The bar for Kristen’s data and Ricardo’s data are incorrectly positioned. Should this response receive partial credit? If so, how much? Should the student receive 2 points for this item?
Example 6: Bar Chart Partial Credit?
31
Presenter
Presentation Notes
In this response the student has correctly positioned 1 of the 4 bars. The bar for Nancy’s data is positioned correctly, but the other 3 bars are not. Should this response receive partial credit? If so, how much? Should the student receive 1 point for this item?
Example 7: Line Graph
32
Presenter
Presentation Notes
This is a line graph item in which students are asked to graph the data in the table. Please take a moment to read the question and think about your response.
Example 7: Line Graph Full Credit
33
Presenter
Presentation Notes
This is the correct response which would receive full credit? Are there responses you think should receive partial credit?
Example 7: Line Graph Partial Credit?
34
Presenter
Presentation Notes
In this response the student has plotted 5 of the 6 points correctly. Only the last point (for time=10) is plotted incorrectly? Should this response be awarded partial credit? If so how much? Should this response receive 5 points? Half credit?
Example 7: Line Graph Partial Credit?
35
Presenter
Presentation Notes
In this response the student is off by a value of 1 for the y-axis (volume) for each plotted point. However, the shape of the line is correct—just shifted down by 1 unit. Should this response receive partial credit?
Example 8: Fill-in-the-Blank
36
Presenter
Presentation Notes
This is an example of a fill-in-the-blank item. Take a moment to read the item and think about your response
Example 8: Fill-in-the-Blank Full Credit
37
Presenter
Presentation Notes
Here is the correct response which would be awarded full credit. Note that there is only one response area and so would not be eligible for partial credit scoring as a result.
Take-Aways 1) Are there some items for which partial credit
should not be offered even when there are multiple responses?
2) Are there some response patterns which should receive partial credit and others which should not?
3) How many points of partial credit should each response pattern be worth?
4) Does order of responses matter in awarding partial credit?
