holistic scoring in social studies curriculum, instruction, and instructional technology team new...
TRANSCRIPT
Holistic Scoring in Social Studies
Curriculum, Instruction, and Instructional Technology Team
New York State Education DepartmentAlbany, NY
2005
NYS Education Department 2005
Contact Information
Curriculum & Instruction
Jo Ann Larson
Lawrence Paska
Phone: (518) 474-5922
State Assessment
Donna Merlau
Gary Warren
Phone: (518) 474-3860
NYS Education Department 2005
Purposes of the Regents
• Accountability: To measure student attainment of the Learning Standards for Social Studies at the commencement level
• Diagnostic: To help school districts identify those students who are in need of Academic Intervention Services (AIS)
NYS Education Department 2005
Goal
• To achieve consistent and accurate scores
Note: not simply to have everyone assign the same scores,
but to assign them for the right reasons.
NYS Education Department 2005
Purpose of This Workshop
• Make the rating task as simple and painless as possible
NYS Education Department 2005
Cast of Characters
PRINCIPAL / SOCIAL STUDIES COORDINATOR
• Appoints a Scoring Coordinator who will manage the training and logistics of the scoring process
• Appoints a Trainer
NYS Education Department 2005
Cast of Characters
SCORING COORDINATOR• Manages the training and logistics of scoring• Organizes the movement of papers throughout
the entire scoring process• Records scores on Part I answer sheets
NYS Education Department 2005
Cast of Characters
TRAINER• Responsible for the overall training session,
including time management (starting on time, moving from step to step of training procedures, keeping to schedule)
• Responsible for the substance of scoring• Leads review of: specific rubric, procedures for
assigning holistic scores, each anchor paper and commentary, practice papers
NYS Education Department 2005
Cast of Characters
TRAINER• Follows the procedures and adapts them when
necessary to ensure that everyone can assign valid and consistent scores
• Answers questions that arise over how to evaluate papers or assign scores
• Monitors scoring session progress; helps keeps Raters on schedule
• Assists the Table Leaders with back-reading
NYS Education Department 2005
Cast of Characters
TABLE LEADERS• Distributes the bundles of papers to be scored
(with Support Staff help)• Leads review of the generic rubric and specific
rubric with reference to the task by matching evidence from the student responses to the rubric
• Leads review of each anchor paper and commentary
NYS Education Department 2005
Cast of Characters
TABLE LEADERS• Leads discussion until Raters feel confident to
move on to actual rating; discusses any discrepancies
• Helps the Trainer with the substantive issues that arise during discussion (e.g., why a paper received a certain score, how to explain a distinction between score levels)
NYS Education Department 2005
Cast of Characters
TABLE LEADERS• Back-reads several papers for each scorer, to
confirm that Raters assign scores consistently• Serves as a third Rater to resolve scores that
differ by more than one point• Calculates a final score for each student’s essay• Ensures that Raters fill out score sheets correctly• Assembles all score sheets and training materials
at end of scoring session
NYS Education Department 2005
Cast of Characters
RATERS• Scores the five practice papers without looking at
the scores and commentaries provided• Records own individual rating for each student’s
essay on the rating sheet provided (not directly on the student’s paper/answer sheet), when the actual rating begins; the scores of each scaffold question may be recorded in the student’s examination booklet
NYS Education Department 2005
Cast of Characters
RATERS• Should not correct the student’s work by making
insertions or changes of any kind• Note: Each essay must be scored by two
different Raters; scaffold questions only need scoring by one Rater
NYS Education Department 2005
Cast of Characters
RATERS: QUALIFICATIONS• Teachers of Grades 7-12 social studies• Special Education Teachers who are
knowledgeable about the GHG or USHG social studies curriculum
• Should have previous school, district, and/or regional training on scoring social studies essays or scaffold questions as part of the turnkey training process
NYS Education Department 2005
Cast of Characters
SUPPORT STAFF• Distributes the bundles of papers to the tables• Moves the papers so that all parts and all papers
are completely scored• Collects the score sheets and training materials,
moving them to shipping centers or a storage facility, at the end of the session
NYS Education Department 2005
The Rating Process
1. BEGIN• Select a Table Leader.• Review each scaffold question and/or essay task.• Discuss possible student responses.• Review and discuss the specific rubric.• Discuss the nature of responses that would be
expected at each score point of the rubric.Remember: This is a holistic scoring. A student does not need to achieve a “5” for each bullet in the rubric in order to be awarded a “5” on the essay. A “5” is not necessarily a perfect paper.
NYS Education Department 2005
The Rating Process
2. REVIEW THE SET OF ANCHOR PAPERS• Read the student anchor papers individually.• Starting with the first anchor paper, the group
should discuss the scores that were awarded. Using the specific rubric, explain why each anchor paper earned the score that it did.
• Read the commentaries at the back of the packet for each of the anchor papers, to find the rationale for each paper’s score.
NYS Education Department 2005
The Rating Process
3. REVIEW THE PRACTICE PAPERS• Read the student practice papers individually.• Using the specific rubric, each Rater at the table
awards a score to each practice paper without looking at the scores and commentaries provided after the papers.
• After each Rater has separately scored the practice papers, the Table Leader charts the scores and leads discussion on the scores that were awarded.
NYS Education Department 2005
The Rating Process
3. REVIEW THE PRACTICE PAPERS• Where there are discrepancies, the Table Leader
should ask individuals with the most discrepant scores to discuss their rationale for awarding credit. The group should continue to discuss the scores until consensus is reached. (See the commentaries at the back of the packet for the scores awarded for each practice paper.) Discuss any discrepancies between the SED commentaries and the scores awarded above.
NYS Education Department 2005
The Rating Process
3. REVIEW THE PRACTICE PAPERS• If additional practice is required to reach scoring
consensus, a sample of actual student answer papers from the test may be used.
NYS Education Department 2005
Setting the Scene …
• Provide frequent breaks.• Provide incentives (e.g., water, snacks, meals).• Conduct the scoring session in a pleasant
environment.• Check for understanding of rating procedures
frequently.• Review the rubric after any long-term break
(e.g., a lunch period, overnight).• Work toward exact agreement in awarding credit.
NYS Education Department 2005
Materials Needed
• Photocopiers• Fax machines• Computers and printers• Paper• Chart paper• Paper clips and rubber bands• Pens, pencils, markers, and highlighters
NYS Education Department 2005
Checking for Accuracy
• Do the Raters share a common interpretation of the rubric, or do some Raters have their own personal construct of the rubric?
• Are Raters applying the rubric appropriately, or are they influenced by irrelevant factors when scoring student performance?
• Are Raters using all of the criteria, or are they ignoring some aspects?
• Are Raters assigning scores reflective of the rubric?
NYS Education Department 2005
Other Reminders
• “Scoring Key and Rating Guide” may not be used until after the Uniform Statewide Admission Deadline (45 mins. after exam start time).
• Set aside one room as a central rating room for collecting, storing answer sheets/essay booklets, and for preparing/maintaining records, for the exam.
• Avoid having a teacher rate his/her own students’ responses.
• Provide adequate time for training and for rating.