grading and analysis report for clinical portfolio 1

Grading and Analysis Report

For Clinical Portfolio

1

Analysis for Formal Traditional Assessment

All grades and names recorded in this example are from the accompanying handouts.

1. After grading, Rank order the students according to overall grade.

2. Record the results of each item in a chart like your worksheet and the following slide.

1 is used to indicate a correct answer and a 0 is used to indicate an incorrect answer.

If a student were to earn half credit for an item, a 0.5 would be used. 2


Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 Julio 1 1 1 1 1 5/5 LaShae /5 Thomas /5 Jean /5 John /5 Becky /5 Ellen /5 Juan /5 Σ n= p value ng = D value

3

Analysis for Formal Traditional Assessment 3. Once all scores are recorded for each student in

each item, compute the sum for each item (sum the column for each item).

4

Analysis for Formal Traditional Assessment Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 Julio 1 1 1 1 1 5/5 LaShae 1 0 1 1 1 4/5 Thomas 1 0 1 1 1 4/5 Jean 1 1 0 1 1 4/5 John 1 1 0 1 1 4/5 Becky 1 1 0 0 1 3/5 Ellen 1 1 0 0 1 3/5 Juan 0 0 0 1 1 2/5 Σ 8 n = 9 p value ng = D value

5


6

Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 Julio 1 1 1 1 1 5/5 LaShae 1 0 1 1 1 4/5 Thomas 1 0 1 1 1 4/5 Jean 1 1 0 1 1 4/5 John 1 1 0 1 1 4/5 Becky 1 1 0 0 1 3/5 Ellen 1 1 0 0 1 3/5 Juan 0 0 0 1 1 2/5 Σ 8 6 4 7 9 n = 9 p value ng = D value


4. Next compute the p value (item difficulty index) for each of the items using the equation below.

All answers will range from 0.0 to 1.0.

p = # of students who answered the item correctly -----------------------------------------------------

Total # test takers

OR p = Σ (columnx) ---------------

n7


8

Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 Julio 1 1 1 1 1 5/5 LaShae 1 0 1 1 1 4/5 Thomas 1 0 1 1 1 4/5 Jean 1 1 0 1 1 4/5 John 1 1 0 1 1 4/5 Becky 1 1 0 0 1 3/5 Ellen 1 1 0 0 1 3/5 Juan 0 0 0 1 1 2/5 Σ 8 6 4 7 9 n = 9 8/9= p value .89 ng = D value


9

Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 Julio 1 1 1 1 1 5/5 LaShae 1 0 1 1 1 4/5 Thomas 1 0 1 1 1 4/5 Jean 1 1 0 1 1 4/5 John 1 1 0 1 1 4/5 Becky 1 1 0 0 1 3/5 Ellen 1 1 0 0 1 3/5 Juan 0 0 0 1 1 2/5 Σ 8 6 4 7 9 n = 9 8/9= 6/9= 4/9= 7/9= 9/9= p value .89 .67 .44 .78 1.0 ng = D value


p Value Interpretation for Classroom Teachers:

A p value below 0.75 is considered problematic. That tells us that at least 25% of the students answered the item correctly…

That could mean that: 1) the students do not understand the information tested by

the item; 2) the item is not well written, but the students do understand

the information; 3) the item is miss-keyed or there is more than one right

answer; 4) the students did not study.

10


5. Divide the students into 3 groups, based on their grades.

Put the upper 1/3 in group 1(g1), the middle 1/3 in group 2(g2), the lower 1/3 in group 3 (g3).

Groups 1 and 3 must have an equal number of students in the groups.

11


12

Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 G1 Julio 1 1 1 1 1 5/5 G1 LaShae 1 0 1 1 1 4/5 G1 Thomas 1 0 1 1 1 4/5 G2 Jean 1 1 0 1 1 4/5 G2 John 1 1 0 1 1 4/5 G2 Becky 1 1 0 0 1 3/5 G3 Ellen 1 1 0 0 1 3/5 G3 Juan 0 0 0 1 1 2/5 G3 Σ 8 6 4 7 9 n = 9 8/9= 6/9= 4/9= 7/9= 9/9= p value .89 .67 .44 .78 1.0 ng = D value


6. Compute the D values (item discrimination index) for the items using the formula below. Notice the difference in the denominator of this equation.

All answers will range from -1.0 to 1.0.

D =# correct upper group - # correct lower group ------------------------------------------------------ Total # test takers in each group

ORD = Σu – Σl

--------------- ng 13


14

Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 G1 Julio 1 1 1 1 1 5/5 G1 LaShae 1 0 1 1 1 4/5 G1 Thomas 1 0 1 1 1 4/5 G2 Jean 1 1 0 1 1 4/5 G2 John 1 1 0 1 1 4/5 G2 Becky 1 1 0 0 1 3/5 G3 Ellen 1 1 0 0 1 3/5 G3 Juan 0 0 0 1 1 2/5 G3 Σ 8 6 4 7 9 n = 9 8/9= 6/9= 4/9= 7/9= 9/9= p value .89 .67 .44 .78 1.0 ng = 3 (3-2)/3 D value .33


7. Determine which items are problematic according to p and D values.

15

Item 1 2 3 4 5 Grade Group Name Brian 1 1 1 1 1 5/5 G1 Julio 1 1 1 1 1 5/5 G1 LaShae 1 0 1 1 1 4/5 G1 Thomas 1 0 1 1 1 4/5 G2 Jean 1 1 0 1 1 4/5 G2 John 1 1 0 1 1 4/5 G2 Becky 1 1 0 0 1 3/5 G3 Ellen 1 1 0 0 1 3/5 G3 Juan 0 0 0 1 1 2/5 G3 Σ 8 6 4 7 9 n = 9 8/9= 6/9= 4/9= 7/9= 9/9= p value .89 .67 .44 .78 1.0 ng = 3 (3-2)/3 (2-2)/3 (3-0)/3 (3-1)/3 (3-3)/3 D value .33 0.0 1.0 .67 0.0


D Value Interpretation for Classroom Teachers:

D values are really something that Norm referenced test writers are concerned with.

• Typically, any item with a D value below 0.40 or negative in value is considered to not discriminate between those with the highest grades and those with the lowest grades.

Really, in the classroom, we want all students to do well!!, so typically we REALLY want the students with the lowest grades to correctly answer as many items as possible!

16



Classroom teachers are concerned with an item that has a D = or approaches 1.0

• Means that all of the top/highest scoring group (G1) answered the item correctly and all or almost all of the bottom/lowest group (G3) answered the item incorrectly.

Tells us that at least the lowest 1/3rd of the class has missed the concept of that item completely

…Definite need for remediation.

17



Also concerned with any item that has a D = some negative value,

• Means that more of the students with the lowest scores answered correctly than those with the highest scores.

First and foremost, the item has to be checked to see if it is miss-keyed.

Negative value could also be the result of • poor directions; • poor item construction; • or the common occurrence of those who score well to

typically rush through assessments and not read directions or items completely. 18

Analysis for Formal Alternative Assessment

Analysis of the function of the Grading RubricHow well does the Rubric actually grade the

product/presentation, the students’ knowledge and skill capability?

1. After grading, Record the score received from each criterion for each student in a chart like the one on the next slide. • This will be done for each criterion that you

measure with the rubric.

• An X indicates the points earned by each student for the criterion.

19

Analysis for Formal Alternative AssessmentCriteria 1: Aesthetics

5 4 3 2 1 Name Jean X Becky X Juan X John X Brian X David X Henry X Julio X Thomas X LaShae X Julie X Σ n = (100) Frequency

20


2. Once all scores are recorded for each student in each criterion, calculate the sum for each column for each point value.

3. Next compute the relative frequency of each score by dividing the sum of each column by the total number of students.

4. To create a number that is easier to interpret, multiply frequency decimal by 100. You will do this for each point value for each criterion.

21


5 4 3 2 1 Name Jean X Becky X Juan X John X Brian X David X Henry X Julio X Thomas X LaShae X Julie X Σ 3 2 3 2 1 n = 11 3/11 = 0.27 2/11 = 0.18 (100) 0.27(100) 0.18(100) Frequency 27% 18%

22


5 4 3 2 1 Name Jean X Becky X Juan X John X Brian X David X Henry X Julio X Thomas X LaShae X Julie X Σ 3 2 3 2 1 n = 11 3/11 = 0.27 2/11 = 0.18 3/11 = 0.27 2/11 = 0.18 1/11 = 0.09 (100) 0.27(100) 0.18(100) 0.27(100) 0.18(100) 0.09(100) Frequency 27% 18% 27% 18% 9%

23


5. Once this has been done for each of the criteria, analyze the frequencies to determine how appropriate the criteria graded the performance assessment.

For the previous chart,Aesthetics, 72% of the students scored a 3 and above, and 27% of the students scored a 1 or

2. The missing 1% is due to rounding errors.

This criteria is an appropriate measure of the aesthetics of the student work b/c more than 70% of the students scored

a 3 or higher (meaning that at least 70% of the students scored a D or higher on that criteria).

24


If the percentage of 3 or higher is below 70%, then the criteria • may not be grading effectively because it does not match the

expectations; • or is grading too harshly against the quality of work the

students are capable of.

If either is true, the criteria would need to be adapted before final grades are returned to the students.

It may also mean that the students did not work up to their full potential, in that case the rubric can stand, but the responses must be returned with extensive constructive feedback.

My analysis of the Aesthetics criteria is that it is grading the aesthetic quality of student work effectively b/c 72% of the student scored a passing grade or higher. 25


6. Do this analysis for each criterion and based on the results, determine if the entire rubric is appropriate or not.

This is done by looking at the analysis of each criterion.

If more of the criteria are effective than ineffective, the rubric is an effective grading tool.

26

Analysis for Formal Alternative AssessmentCriteria 2: Content

10 8 7 2 Name Jean X Becky X Juan X John X Brian X David X Henry X Julio X Thomas X LaShae X Julie X Σ 3 3 4 1 n = 11 3/11 = 0.27 3/11 = 0.27 4/11 = 0.36 0/11 = 0.00 (100) 0.27(100) 0.27(100) 0.36(100) 0.00(100) Frequency 27% 27% 36% 0%

27

Analysis for Formal Alternative AssessmentCriteria 3: Presentation

5 4 3 2 Name Jean X Becky X Juan X John X Brian X David X Henry X Julio X Thomas X LaShae X Julie X Σ 4 3 1 3 n = 11 4/11 = 0.36 3/11 = 0.27 0/11 = 0.00 4/11 = 0.36 (100) 0.36(100) 0.27(100) 0.00(100) 0.36(100) Frequency 36% 27% 0% 36%

28

Grading and Analysis Reportfor the Clinical Portfolio

See Wiki for Example of Grading and Analysis Report

For Traditional Assessments- Create a chart like the one on slide 3- Fill in your information- Complete the statistics to find the p and D values- Answer the following questions

- Which items fall below the p value of 0.75? Why do you think this occurred? What would need to be done about these items?- Which items have a D value of 1.0 or a value that is negative? Why do you think this occurred? What would need to be done about these items?

29

Item Analysis for the Clinical Portfolio

For Performance Assessments- Create a chart like the one on slide 20 for each criteria

(maximum of 4)- Fill in your information- Complete the statistics to find the frequency values- Answer the following questions

- How do each of the criteria rate? How does this overall rubric rate? What, if anything, would need to be done before this rubric is used for grading?

30

Final Reflection

31

A reflection commenting on the assessment’s effectiveness is required after you analyze student responses. This is a brief reflection (max 1 page) discussing the experience of administering your traditional and alternative assessments. Please discuss good and bad items/experiences, suggestions for changes, and theories for why certain items/techniques did or did not work as well. Also include how the student responses reflected what was taught following the assessment.

grading and analysis report for clinical portfolio 1

Documents