+ monitoring access tools and accommodations in a new assessment context laurene christensen, ph.d....
TRANSCRIPT
+
Monitoring Access Tools and Accommodations in a New Assessment Context
Laurene Christensen, Ph.D.National Conference on Student Assessment
+What is NCEO?
Federally-funded technical assistance center
Support SEAs and consortia on issues related to inclusive assessmentStudents with disabilitiesELLsELLs with disabilities
Website: nceo.info
+Changing Assessment Context
Common Core State Standards
PARCC AssessmentDLM Assessment
College and Career Readiness
ELPA21
Smarter Balanced Assessment
NCSC Assessment
ASSETS
+ From Accommodations to Accessibility Tools and Accommodations
Accommodations are adjustments to the testing situation, test format, or test administration that provide equitable access during assessments for students with disabilities and students who are English learners.
+Smarter Balanced Framework
+PARCC Framework
+Accommodations on DLM
Dynamic Alternate Assessment System
Universal design
Variety of response formats
Compatible with variety of assistive technologies
Allows for varying levels of teacher assistance
+Accommodations on NCSC
Format of assessment (AA-AAS): 4 complexity levels; 30 items per subject; can
be delivered online Most items are multiple choice Some constructed response; can use picture
symbols or graphic organizers
Most accommodations are embedded within items
Allowed accommodations are included in test administrators’ scripts
+Accommodations on ELPA21
Still being decided
Administrative Considerations
Universal Features
Designated Features
Accommodations
+Accommodations on ASSETS
ELP assessments
Computer delivered
Accommodations and accessibility features built-in
Designed to be appropriate for a range of needs
+How do all of these pieces come together?
+First steps: The Need for Monitoring
Monitoring requirements of IDEA
Monitoring requirements from ESEA
Expectations from peer review (uncertain right now)
+What is monitoring accommodations?
Monitoring accommodations is a process of evaluating policies, procedures, and practices in order to ensure that a state’s assessment system is inclusive of students with disabilities and English language learners.
+Monitoring is about compliance
The State has analyzed the use of specific accommodations for different groups of students with disabilities and has provided training to support sound decisions by IEP teams.
The State routinely monitors the extent to which test accommodations are consistent with those provided during instruction.
But it is about more than compliance….
+Monitoring is about improving outcomes
Systematic attention to the provision of accommodations can ensure that students are able to show what they know and can do
+Monitoring Accommodations: Overview of Tool
Developed in collaboration with CCSSO Reviewed extensively by ASES SCASS group Based on a review of publicly available
materials 5 Step process Each step includes examples, a checklist,
additional resources Appendices include additional examples
from states Extensive glossary at the end
+5 Steps to Monitoring Accommodations
1. Know the rules and regulations for accommodations monitoring
2. Document decisions about access tools and accommodations
3. Document the use of access tools and accommodations
4. Review access tool and accommodations decisions and use
5. Evaluate and report on access tools and accommodations
+Document Decisions about Accessibility Tools and Accommodations
Keeping track of training
Keeping track of decisions General ed students ELLs Students with disabilities
Are access tools and accommodations used for instruction consistent with those used for assessments?
Keeping track of requests for unique accommodations
+Document the use of accessibility tools and accommodations
Documenting on the PNP/ISAAP
Documenting inappropriate use of tools and accommodations on test day
Documenting that students received tools and accommodations on test day
+Review accommodations decisions and use
Direct observation on test day
Online record reviews
On-site visits on a day other than test day
Interviews with students, teachers, and administrators about the use of accommodations
+Evaluate and Report on Accommodations
Analyze accommodations
Report on accommodations
Revise accommodations policies
+Questions to Consider
How can we support educators in keeping up with the different policies of each assessment?
What additional support is needed to effectively implement the PNP?
With all of the data these new systems can generate, what are the questions we should ask of the data?
Beyond the PNP:Designing a Comprehensive State System
for Systematically Monitoring Accommodations
Melissa Gholson, CoordinatorOffice of Assessment and
Accountability
Common Core EssentialElements
Our System
English Language Proficiency Standards
2014 and Beyond
• Stakeholders meeting• Reviewing current policies, processes and
procedures• Develop short and long term goals• Supporting an ongoing research plan• Focus on improving our system
A Systems Perspective
A Systems Perspectives
Research Agenda
Peer Review
Participation Guidelines & Policies
WVEIS Data Process, Codes & Descriptions
IEP, 504, SAP and LEPPlans
Monitoring and the WVS 326 Process
Improving Accommodations
• Ensure that your state has a valid method for gathering data on which students are to receive specific accommodations in assessment, and a form to document what they receive on test day. (Electronic WVS.326 Process)
• Document how your state analyzes its accommodations data, including timelines ofanalysis. (State & District Reports)
• Establish a specific monitoring procedure that identifies issues in the selection of accommodations for individual students or the provision of accommodations for instruction or assessment. (Participation Guidelines)
• Include information on any consequences that result from any irregularities in the administration of assessment accommodations. (Report analysis, investigation team and invalidation procedures)
• Conduct studies that examine the link between instructional accommodations and assessment accommodations (Research agenda)
Accommodations Research Plan
2009Paper
version of
monitoring
2010Pilot
Electronic WVS
326ACT
2011Seconda
ry Feasibilit
yW2
2012All WV-MapsELL
Research Developm
ent
2013State & District
Reporting
2014 and Beyond
Focus on improvin
g the process
Accommodations Research – Using last 3 years of W2 data– Will be able to look for trends– The primary purpose of this report is to describe the accommodations assigned to
SWD, LEP, and 504 students for the 2011 – 2013 school years. • Discussion/Interpretation: How appropriate are the accommodation assignments based upon
student subgroup/performance?• How do these trends vary across time?
– Report may be able to - Describe: Relationship of WT2 to Accommodation Assignments
• Does WT2 performance predict accommodation assignments? 2010 assessment and 2011 accommodations
– Do accommodation assignments influence WT2 performance? 2010 accommodations and 2011 assessment
– Examine 2013-2014 online W2 data independently– After 2-3 years may be able to do DIF– Continue analyzing ELL data by ELP proficiency, achievement, by years in program,
include ELL who have met exit criteria for two years.
Accommodation Code Accommodation Shorthand n of Students Recommended to Receive Accommodation
% of Sample Recommended to Receive Accommodation
P02 Have test read aloud: Math, Science, SS Tests
13,456 58.8%
T04 Use extra time (not applicable to WESTEST 2)
6,750 29.5%
P26 Have directions/passage and prompt read aloud -- on-line writing
6,472 28.3%
P15 Have directions only read aloud--RLA
5,686 24.8%
T03 Provide more breaks 5,085 22.2%P18 Have directions rephrased by
trained examiner4,381 19.1%
T07 Flexible scheduling 4,104 17.9%R02 Indicate response to a scribe
(selected response and on-line writing)
499 2.2%
R04 Indicate response to a scribe (constructed response)
445 1.9%
P19 Use a large-print edition of the test
139 0.6%
Most Frequent Accommodations for Students with Disabilities
Peer Review Requirements • Providing an appropriate variety of assessment accommodations
• Ensuring that the use of assessment accommodations yields meaningful scores
• Ensuring that the use of accommodations is consistent with instructional approaches, as determined by a student’s plan
• Determining that assessment scores for students when administrated under accommodated conditions, allow for valid inferences about the students’ knowledge and skills
• Establishing clear guidelines for including all students in assessment
• Differentiate between accommodation policies for students with disabilities and English Language Learners.
Participation Guidelines
• The document includes references to current state policies and procedures.
• The document is updated and revised annually by a stakeholder group
Accommodations Review
Smarter Balanced
Smarter Balanced
Universal ToolsAvailable to all
Designated SupportsAvailable for Students with LEP, IEP and 504 Plans (ISAAP)
AccommodationsAvailable for Students with IEP and 504 plans.
EmbeddedInto the platform
BreaksCalculatorDigital NotepadEnglish DictionaryEnglish GlossaryExpandable PassagesGlobal NotesHighlighterKeyboard NavigationMark for ReviewMath ToolsSpell CheckStrikethroughWriting ToolsZoom
Color Contrast/P28Masking /P37Text-to-Speech /P01Translated Test Directions/P30Translations(Glossary)/P31Translations (Stacked)/P32Turn off Any Universal Tools/P33
American Sign Language/P34Braille/P03 or P35Closed Captioning/P36Text-to-Speech/P13
Non-EmbeddedProvided locally
BreaksEnglish DictionaryScratch PaperThesaurus
Bilingual Dictionary /R15Color Contrast/P38Color Overlay/P39Magnification/P23Read Aloud/P02Scribe/R04Separate Setting/T09Translations(Glossary)/P41
Abacus/R05Alternate Response Options/R11Calculator/R19Multiplication Table/R20Print on Demand/P40Read Aloud/P14Scribe/R04Speech-to-Text/R21
Accommodations ProcessDecision to provide an accommodation is documented in the student plan.
The documentation is entered into the WVEIS data system
Data must be verified and
reviewed prior to assessments
Accommodations Application
provides the data on a daily delta
Accommodations data is collected analyzed, reported
to districts and used to inform policies, practices and
provide targeted technical assistance.
Accommodations Electronic Application
• Accommodations application supports the process for identifying students who need accommodations
• Accommodations are updated on a daily delta and are available at the school, district and state level.
• The accommodations application allows users to filter and sort to by school, student ID, group, name, grade or accommodation.
• The application allows for printing and the creation of spreadsheets
Verification Process
• Accommodations are verified locally prior to assessment.
• WVEIS Accommodations App are compared to current student plans
• The WVS. 326 is reviewed to prepare the testing environment for the student.
WVEIS Data System:Accommodations Application
Data Sorting &Filtering
The WVS.326 Process:Accommodations Provision
and Monitoring
• The WVS.326 form is provided for all state and district assessments.• One form for each student receiving identified supports and
accommodations.• Electronic process which feeds accommodations research
information to support better student outcomes.• Accommodations data collected informs accommodations policies,
practice and research agendas.• The WVS.326 process provides feedback on accommodations
provision for student plans.• Documentation of the WVS. 326 form is available http://
wvde.state.wv.us/oaa/videos/WV_326/Documentation%20Procedures.zip
WVS.326
• The WVS.326 form includes all the accommodations
• All pre-slugging and data entry one side
• Directions are on the back• Coding process includes data
for accommodation refusals and is used for invalidations of over and under accommodations.
• Information about WVS. 326 process is included in the Participation Guidelines.
Let’s Talk About CodesIf you mark Provided “Yes”- this means the student was given the identified accommodation.If you mark “No”- this means the student was not provided the identified accommodation. This is an accommodations breach UNLESS it is a:• Code 1-the student refused; or• Code 2-the accommodation is not allowed/applicable for
this testIf “no” is not marked with a code 1 or 2, then it constitutes an accommodations error that requires documentation. See the PG for specific details.
What to do about students who do not receive accommodations?
All incidents for failing to provide an accommodation requires written documentation.
• The CTC will review all accommodation breaches with principals/building level coordinators and report any findings to the Special Education, Title III and/or 504 director(s) and to the Office of Assessment and Accountability.
• If a student has been denied accommodations listed within the IEP, 504 or LEP plan, there has been a breach in the integrity/accuracy of test results. Therefore, the CTC or county special education director should contact the principal who must inform the parent/guardian of the testing administration breach and provide the following options:
– Test is invalidated and the student is retested using a breach form (if the testing window is still open). – Test is scored. – Test is invalidated and student receives a score of ―0.
If the second or third option is selected—there must be a signed written agreement between the parent/guardian and principal/building level coordinator. Copies of the agreement must be kept on file with the CTC/county special education director, Section 504 director, or Title III director.
WVS 326 State and District Accommodations Report
• Analyzed provision data for 2012 WESTEST 2 by content area• 7 reports were generated for each content area to
summarize (1) provision rate, (2) refusal rate, (3) not allowable rate, and (3) number of students over-accommodated– State, district, school-level reports – Intended uses of these data– Set specific targets for provision rates– Identify common errors to inform your training– Eliminate under- and over-accommodation
WVS. 326Accommodations Reporting
1. Count of Students with Accommodation
This report provides the total number of students with the accommodation listed on the front of the 326 form (this number serves as the denominator for all rate calculations in reports 2 – 6).
2. Provision Rate
This report provides the percentage of students with the accommodation for whom the accommodation was marked “provided.”
3. Not Allowed Rate
This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” and the “not allowed” code was provided in the reason field.
4. Refusal Rate
This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” and the “refused” code was provided in the reason field.
5. Not Provided (no reason indicated)
This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” but no reason code was provided.
6. Not Provided (blank)
This report provides the percentage of students for whom no information was provided on whether or not the accommodation was provided.
7. Count of Students Over-accommodated
This report provides the number of students who do not have the accommodation listed in the pre-slug file, but for whom the accommodation was marked “provided.”
State Reports
District Reports
Lessons Learned During the Transition
• Collaboration is critical. • Review and update WVEIS codes and descriptions in all policy and practice documents.• The codes we use for a PNP to identify specific students’ needs for example, braille, large print,
etc. will correspond to the accommodations data map that must be provided to test vendors.• The Participation Guidelines must be updated and cover the parameters of the designated
systems such as TIDE and define the allowable universal tools designate supports and accommodations .
• OAA must provide training to districts, including CTC’s, Special Education Directors, 504 Coordinators, ELL Coordinators, principals/BLC, special education teachers, student assistance teams, teachers and test examiners.
• The LEP, 504, SAP Plan & the Online IEP system must be reviewed and incorporate available tools, designated supports and accommodations.
• Communicate plan changes to all impacted, including parents.• Student plans must be reviewed, amendments made or meetings held in order to address the
changes.• The WVS.326 form must be adjusted, a new program created for pre-slugging and reading the
form to provide monitoring data for accommodations
54
Beyond the PNP:Designing a Comprehensive State System for Systematically Monitoring Accommodations
Brian M. TouchetteDirector
Office of Assessment
55
State Profile• 3 counties
o 19 districtso 25 charterso ~220 schools
• 130,000 total students (15,000 w/ IEPs)• 80,000 students assessed (10,000 w/ IEPs)• DE transitioned from paper based assessment to
on-line assessment 2008-09 school year
56
On-line IEP• State mandated form• Accom. Form included• Form is printed for
student’s audit File• State can view any
student’s IEP and attached forms
• Universal design vs. accommodations
57
Accommodations Data-baseOn-line IEP
• Paper version of Accom. Form is entered
• Separates accoms. for each content area
• Often entered by office admin.• Functional vs. Non-functional
58
Functional vs Non-Functional Accommodations• Functional Accommodations
o These are provided within test delivery system to student during online assessment
o These include text-to-speech, font size (can be pre-set), print-on-request, color choices (available to all)
• Non-Functional Accommodationso These accommodations are provided to student by test
administratoro These include: human reader, external mouse (available to
all), special equipment
59
Universal Design vs. Accommodations
• Some universal design features are also included on Accommodation Form
• UD features still available even if IEP team did not select
• Way to ensure students with IEPs still received• UD features not counted as accommodation in
federal or state reporting
60
Accommodations Data-base
Accommodation Data Sent to
Vendor
On-line IEP
• Functional and Non-Functional are sent
• Daily upload• Accoms can change
during a students test (may take up to 72 hours to sync)
61
Review of Assessment Accommodations in DCAS TIDE • Review DCAS TIDE for
accommodationso Review
accommodations within TIDE
o Accommodation summary displayed next to each student name
62
Functional Assessment Accommodations in DCAS TIDE • Review individual student
accommodationso Review functional
accommodations within TIDEo Select VIEW button next to a
student name.
63
Non - Functional Assessment Accommodations • Non-Functional
Accommodationso Review non-functional
accommodations within TIDEo Select VIEW button next to a
student name. o This will display all of the
accommodations assigned to student.
64
Student Accommodations Review• Validate all accommodations for testing • Make sure all new IEP accommodations have been
modified or added 3 days before testing• Please provide a hard-copy of student
accommodations for Test Administrator to validate when in the test administration window
• Additional accommodation information can be found in the Test Administration Manual (TAM)
65
Accommodations Data-base
Test Session
Accommodation Data Sent to
Vendor
On-line IEP
• Mandatory PD • All accoms must be
provided, available and accessible during test
• Accoms appear on Test Administrator’s screen– Must verify both functional
and non-functional
66
Test Administration Session – Validate Accommodations• If student has accommodations, select the See/Edit
details button
67
Test Administration Session – Validate Accommodations
• Review Accommodations against Accommodations Form (paper version)
68
Accommodations Data-base
Accommodations Data Returned to
DOETest Session
Accommodation Data Sent to
Vendor
On-line IEP
•Provides accoms at time of test•Long string of information•Does not show if student actually used an accom or not
69
Accommodations Data-base
Accommodations Data Returned to
DOETest Session
Accommodation Data Sent to
VendorMonitoring
On-line IEP
70
Monitoring throughout the System• Test Security:
o From teachers, students and parentso From data forensics o From special education state complaints
• Primarily completed off-site• Random reviews of accommodations used during
testing vs what is in DE data base• Do not currently compare what is on IEP paper
version what is in the system (unless there is complaint)
71
Accommodations Data-base
Accommodations Data Returned to
DOETest Session
Accommodation Data Sent to
Vendor
On-line IEP
Future Change:•New IEP forms for accoms•Does student need additional practice or Preparation?
72
Accommodations Data-base
Accommodations Data Returned to
DOETest Session
Accommodation Data Sent to
Vendor
On-line IEP
Future Change:Approval of Read Aloud (passages) through IEP audit
73
Accommodations Data-base
Accommodations Data Returned to
DOETest Session
Accommodation Data Sent to
Vendor
On-line IEP
Future Change:Direct link: On-line IEP Accoms to DE Accoms Data-base
74
Accommodations Data-base
Accommodations Data Returned to
DOETest Session
Accommodation Data Sent to
Vendor
On-line IEP
Future Change:PD for validating accoms On-site monitoring
75
Accommodations Data-base
Accommodations Data Returned to
DOETest Session
Accommodation Data Sent to
Vendor
On-line IEP
Future Change:Accom data will be sent in way we can delineate accoms
