morphing rubrics to adapt to developmental student needs
DESCRIPTION
Morphing Rubrics to Adapt to Developmental Student Needs. Aja Henriquez, MFA, EdD Student CSUSB English Instructor:Crafton Hills College, California Baptist University. What is a Rubric?. Definition of rubric (ˈruːbrɪk). Why Rubrics?. School Specific Issues. - PowerPoint PPT PresentationTRANSCRIPT
Morphing Rubrics to Morphing Rubrics to Adapt to Developmental Adapt to Developmental Student NeedsStudent NeedsAja Henriquez, MFA, EdD Student CSUSB
English Instructor:Crafton Hills College, California Baptist University
What is a Rubric?What is a Rubric? Definition of rubric (ˈruːbrɪk)
From Collins English Dictionary
1.
a title, heading, or initial letter in a book, manuscript, or section of a legal code, esp one printed or painted in red ink or in some similarly distinguishing manner
2.
a set of rules of conduct or procedure
3.
a set of directions for the conduct of Christian church services, often printed in red in a prayer book or missal
4.
instructions to a candidate at the head of the examination paper
5.
an obsolete name for red ochre
6.
written, printed, or marked in red
School Specific IssuesSchool Specific IssuesApproximately 31% of incoming students
transfer into the developmental level (English 015) at CHC, while 36% transfer into a lower, remedial level course (914): Around 33% of students test into college level English courses (Research Briefs, 2011).
Approximately 52% of students who pass the “Preparation for College Writing” course (English 015) persist in the college and successfully complete a college level writing course (Student Equity Data, 2011).
What We Can ControlWhat We Can ControlThe success rates tell us that there is
something not transferring in our courses.
Many things we cannot control: student readiness/commitment, family problems, financial problems, etc.
We can address our rubrics and how we communicate with the students.
As we know from research, the students may simply misunderstand what we say/write when we grade.
What is the Purpose of a What is the Purpose of a Rubric?Rubric?We use it to grade for a few
reasons:◦To assess specific items in a given
assignment◦To help ensure or increase uniformity
of assessmentIt is a tool for instructors to
communicate with students. Students must be able to
understand the communication.
Language DisconnectLanguage Disconnect We may be taking for granted student
understanding of our rubrics. Developmental courses often act as an
introduction into the academic discourse community (Bizzell, 1982):
“students from different social classes come to school with different abilities to deal with academic discourse: middle-class students are better suited by their socialization in language use to deal with academic discourse's relative formality and abstraction than working-class students are. This unequal removal from academic language is, of course, exacerbated for students whose home language does not resemble the so called standard English” (192).
Student PerspectiveStudent Perspective
VS
“It’s like they [the affluent students] had their own classes, and we [the students from the poor/working class] had our own classes. We were, like, segregated in the same school. “
--Prep for College Writing Student
Demographics of Basic Skills Demographics of Basic Skills StudentsStudents“Crafton Hills College, then,
serves a community where the constituents must commute to work or school, where the poverty and unemployment rates exceed the state average, and where the income is below the national average.”
From Demographics, CHC webpage: http://www.craftonhills.edu/About_CHC/Demographics.aspx
We Should Keep in Mind:We Should Keep in Mind:Students may not understand the
language we use on rubricsWe must ask ourselves if we
have written the rubric for our understanding or student understanding.
Example: Unexamined Example: Unexamined RubricRubric
Excellent Good Adequate Poor Needs WorkControlling Idea:
Organization:
Development:
Audience:
Comments:
A Tool That Doesn’t WorkA Tool That Doesn’t WorkDoes a developmental student
know what these terms mean when it comes to writing?
If the students don’t understand what the rubric means, then they can’t use it to improve.
How can we make this more understandable?
Other ConcernsOther ConcernsEven with some clearer criteria,
there are still some issues with this rubric, which are less simplistic than just the language used.
Validity-does this measure what we want it to measure?
Reliability-are the scores consistent?
Messick (1995)
ValidityValidityContent Validity: to ensure we are
measuring what students should know and not some other thing (personal preferences), we must match each item on the rubric to a course objective.
Substantive Validity: we should make sure that the items on the rubric connect to different types of cognitive processes and are of varying difficulty
Messick (1995)
How to Ensure Content How to Ensure Content ValidityValidity Review the items on your
rubric and make sure they match to the course objectives along with any departmental norms.
English departments often have norming sessions for grading, which these instructors must take into consideration when evaluating the content of their rubrics.
EXAMPLE partial rationalization of content validity
Some things that make it into our grading that should not be there: stapling, niceness of binder or folder, whether it was accompanied with a latte, etc.
Item one corresponds with objectives 4, 5, 6, 11, as well as the grading criteria for writing (demonstrates skillful use of vocabulary and syntax; is generally free from errors in mechanics, usage, and sentence structure).
Item two corresponds with objectives 7, 8, 9, as well as the grading criteria for writing (focuses clearly on the topic and responds effectively to all aspects of the assignment; explores the issues thoughtfully and in depth; is coherently and logically organized with a thesis statement supported by apt reasons and specific, detailed examples).
Substantive ValiditySubstantive Validity
For explanation of Bloom's Taxonomy, watch http://youtu.be/qjhKmhKjzsQ
Ensuring Substantive Ensuring Substantive ValidityValidityFirst of all, we should examine
the items in our rubrics to ensure there is a spread of difficulty.
By doing this, we are able to measure the ability of students who are at different levels ability (low, mid, and high).
Substantive ValiditySubstantive ValidityPERSONS - MAP - ITEMS <more>|<rare> 2 k + | T| | | | m | | | | | j | | | S| 1 + n | | b | f | | i | | | e | h | M|T Q3 o p | |S | 0 c g +M Q2 Q4 | Q1 |S | Q5 |T | | a | S| | | | | | | -1 + | | l | | T| | d | | | | | | | | -2 + <less>|<frequ>
Items do not match the student ability well.
It is difficult to establish which items should be more difficult, since each individual will have personal difficulties.
There should be a greater spread of difficulty.
Rasch ModelRasch ModelThe previous slide showed an
output table from Winsteps, which is based on the Rasch mathematical model.
We don’t all have this program or the ability to use it.
There is a simpler way to examine our student scores using Excel, which most of us already have.
Steps to Examining ScoresSteps to Examining ScoresOpen an Excel pageInput the student scores for each
item and their total score.You do not need to input student
names. Simply using letters will work.
Using Excel, array the scores from highest to lowest.
Examine the scores.Step by step instructions linked on http://www.youtube.com/watch?v=nDez8VmlN9A
Classical AnalysisClassical AnalysisStudent #
Writing Conventions
Integration of Research Format
Content/ Development Organization Total
Points Possible 12 16 12 48 12 100 k 12 16 12 48 12 100 m 12 16 12 46 12 98 j 10 16 12 46 12 96 n 12 12 12 48 9 93 b 9 16 10 44 12 91 f 6 14 10 48 12 90 i 9 12 9 46 12 88 e 10 12 9 44 10 85 h 9 16 9 40 9 83 o 9 12 9 40 9 79 p 9 12 9 40 9 79 c 12 4 3 44 12 75 g 12 8 6 40 9 75 a 9 4 3 40 9 65 l 6 0 4 40 6 56 d 9 0 3 36 6 54
155 170 132 690 160
item variance 3.59 30.36 10.69 12.98 4.13
Pt. B Cor. 0.41 0.93 0.92 0.84 0.82
See Traub (1991) for reliability discussion
Item one does not have consistent scores.
In order to fix this, I would expand this item, Writing Conventions, into its component parts.
Adding more items will help pinpoint the exact issues with writing conventions that give students trouble, which will in turn help me differentiate instruction based on student need.
Fix Through Fix Through MetamorphosisMetamorphosis
Morphed Rubric, Item OneMorphed Rubric, Item One
Performance Levels
Dimensions
Exemplary (4)
Accomplished (3)
Developing (2)
Beginning (1)
Score
Spelling
There are no spelling errors.
Spelling errors are minimal (1/page)
Spelling errors are a problem (2-3/page)
Spelling errors are a problem (4+/page)
/12
Punctuation There are no punctuation or capitalization errors
Punctuation errors are minimal (1 per page)
Punctuation errors are a problem (2-3 per page)
Punctuation errors are excessive (4+ per page)
/12
Fragments There are no fragments in the essay.
Fragments are minimal (1 total)
Fragments are a problem (2 total)
Fragments are excessive (3 or more total)
/12
Run-ons There are no run-ons in the essay.
Run-ons are minimal (1 total)
Run-ons are a problem (2 total)
Run-ons are excessive (3 or more total)
/12
Interpreting the MorphInterpreting the MorphBecause item one was so problematic, I
can interpret that it is not very clear what the students can do to improve, even with explicit in-text notes. (As we may know)
I morphed it by expanding so that both the students and I could understand what wasn’t working in their assignments. ◦They can work on spelling instead of run-ons◦I can work on student weaknesses with
instruction
The Rasch analysis with Winsteps suggests that most of the grade scale is not necessary.
Grading on a 100 point scale suggest the ability to differentiate an essay in 100 ways (Dr. Jesunathadas).
As the Revised Rubric suggests, I would use a 1-4 point scale for each item and then report the final score as a percentage for student understanding.
The changes to the grading scale will help keep the scoring effective (no more half scores that don’t differentiate well).
Hidden Issues with Hidden Issues with RubricsRubrics
Item 2Item 2ITEM NUMBER: 2 Integration of Research
CATEGORY PROBABILITIES: MODES - Structure measures at intersections
P -+---------+---------+---------+---------+---------+---------+-
R 1.0 +0000000000000 6666666666+
O | 000 6666 |
B | 00 66 |
A | 0 6 |
B .8 + 0 6 +
I | 0 4 6 |
L | 0 4 4 6 |
I | 0 4 4 22 6 |
T .6 + 0 4 4 2 2 6 +
Y | 0 4 4 2 2 6 |
.5 + * 4 2 2 6 +
O | 40 42 26 |
F .4 + 4 0 * 62 +
| 4 0 * 6 2 |
R | 4 0 24 6 2 |
E | 4 0 2 4 6 2 |
S .2 + 4 0 *8* 6 2 +
P | 4 0 * * 6 444 2 |
O | 44 8*2 4**44 444* |
N | 444 882200 **4488 ***444 |
S .0 +*************************************************************+
E -+---------+---------+---------+---------+---------+---------+-
-3 -2 -1 0 1 2 3
PERSON [MINUS] ITEM MEASURE
Revised RubricRevised RubricAssignment Rubric: REVISED ESSAY RUBRIC
Performance Levels
Dimensions
Exemplary (4)
Accomplished (3)
Developing (2)
Beginning (1)
Score
Spelling
There are no spelling errors.
Spelling errors are minimal (1/page)
Spelling errors are a problem (2-3/page)
Spelling errors are a problem (4+/page)
/4
Punctuation There are no punctuation or capitalization errors
Punctuation errors are minimal (1 per page)
Punctuation errors are a problem (2-3 per page)
Punctuation errors are excessive (4+ per page)
/4
Fragments There are no fragments in the essay.
Fragments are minimal (1 total)
Fragments are a problem (2 total)
Fragments are excessive (3 or more total)
/4
Run-ons There are no run-ons in the essay.
Run-ons are minimal (1 total)
Run-ons are a problem (2 total)
Run-ons are excessive (3 or more total)
/4
Integration of research
Research is integrated to illustrate and support main points with specific details. At least 3 sources used.
Research is referenced, but specific details are sparse. Only 2 sources used.
Research is referenced, but specific details are not used. 1 source used.
Research is not discussed or referenced, and print out not attached. No sources used
/4
Proper MLA format
Proper margins, font, spacing, header, etc. are used
Proper formatting is followed except for 1-2 items
3-4 formatting mistakes 5+ formatting errors present
/4
Content and Development
The essay adequately contends with the subject matter and offers a clear and concise discussion the prompt. Clear thesis is present.
The essay deals with the subject matter, but fails to address entire prompt. Thesis is present.
The essay does not adequately address the subject matter or the prompt. Thesis is weak or not present
The essay shows a lack of understanding of the subject matter or depth of thought. Attempt at a thesis is completely absent.
/4
Organization
Essay flows and transitions well; Logical paragraphing; Introduction and conclusion present
Essay flows generally well, with only a few choppy areas
Ideas in the essay do not seem to be linked very well. Essay is difficult to understand.
Essay construction impedes the purpose and meaning of the essay
/4
Total /32 *100=
______%
Morphing Your Rubric(s)Morphing Your Rubric(s)Be sure your items are aligned with
objectives and department normsReview your rubric for clear language
and explicit (understandable) description of proficiency levels
Take one set of assignments and array them on Excel to find inconsistent items
Any inconsistent items should be expanded so you can pinpoint what is not working for students
Morph Your InstructionMorph Your InstructionIf you notice there is something on
the rubric that students are consistently unsuccessful at, then you can spend extra time on that in class instead of on items they have already mastered.
The morphed rubric will help you measure whether your instruction is helping or if you need to choose another tactic.
Is it Working?Is it Working?Student scores on each item should be
consistent with their total scores.If not, keep morphing the
rubric/instruction.Look for growth in scores over the
semester, if you are using the same rubric each time. (This also helps students understand consistent expectations)
Keep in mind student lives will occasionally impact their scores over the semester, so don’t freak out if all students don’t show improvement all the time.
Final ThoughtsFinal ThoughtsQuestions?This presentation is linked at
ajahenriquez.wordpress.comA screencast of how to manipulate data in
excel is available at ◦ http://www.youtube.com/watch?
v=nDez8VmlN9A
ReferencesReferencesBizzell, P. (1982). Review: College Composition:
Initiation into the Academic Discourse Community. Curriculum Inquiry, 12(2), 191-207.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist. 50(9), 741-749.
rubric. (n.d.). Collins English Dictionary - Complete & Unabridged 10th Edition. Retrieved October 20, 2011, from Dictionary.com website: http://dictionary.reference.com/browse/rubric
Traub, R. E.; Rowley, G. L., (1991). Understanding reliability. Educational Measurement. 10, 37-45.
Related TitlesRelated TitlesBond, T. and Fox, C. (2007). Applying the
Rasch Model: Fundamental Measurement in the Human Sciences. Lawrence Erlbaum.
Koretz, D. (2008). Measuring up: what education testing really tells us. Cambridge, MA: Harvard University Press.
Ryan, K., & Shepard, L. A. (2008). The future of test-based educational accountability. New York: Routledge.
Spaulding, D. (2008). Program Evaluation in Practice: Core Concepts and Examples for Discussion and Analysis. San Francisco: Wiley.