mcas-alt: alternate assessment in massachusetts technical challenges and approaches to validity
DESCRIPTION
MCAS-Alt: Alternate Assessment in Massachusetts Technical Challenges and Approaches to Validity. Daniel J. Wiener, Administrator of Inclusive Assessment. University of Maryland – Alternate Assessment Conference October 11-12, 2007. - PowerPoint PPT PresentationTRANSCRIPT
MCAS-Alt: Alternate Assessment in
Massachusetts
Technical Challenges and Approaches to Validity
Daniel J. Wiener, Administrator of Inclusive Assessment
University of Maryland – Alternate Assessment Conference October 11-12, 2007
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 2
Participation: Thinking Differently About Who Needs an Alternate Assessment
MCAS-Alt is intended for • Students with significant cognitive disabilities AND• Students who focus on attaining grade-level achievement standards, but who cannot
fully demonstrate knowledge and skills on the test, even with accommodations State has aligned instruction from lowest level of complexity to grade-level
expectations Implications for scoring and reporting results
–Alternate achievement standards–Grade level achievement standards
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 3
Reporting Results Meaningful performance levels reported for MCAS-Alt,
while acknowledging performance is below grade-level expectations
A student can attain real proficiency through the alternate assessment based on grade-level achievement standards
AdvancedProficientNeeds
Improvement
Warning(Failing at Grade 10)
Adv.Prof.Needs Imp.
ProgressingEmergingAwareness
Performance Levels
MCAS Test:
MCAS-Alt:
MCAS-Alt: A “structured portfolio”
Work samples/video/photo evidence (performance), and data charts (progress) are compiled in an annual portfolio
Evidence shows complexity of tasks, and student’s accuracy and independence in performing tasks aligned with required subjects/strands/standardsData
chart
% Accuracy
% Independence
100%
80%
60%
40%
20%
0%12/1/06 12/2/06 12/3/06 12/4/06 12/5/06
4
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 5
Sometimes, It Seems Like This….
Learning Standards
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 6
…It Could Be More Like This…
LearningStandards
Entry Points
Grade 9-10Learning Standard
#10.D.1 for Data &
Statistics:
Use graphical representations and statistics to communicate information about, and compare, sets of data.
Collect, sort, and organize objects by different characteristics
Gather data on familiar objects, and represent them on a simple graph, table, or chart
Compare and contrast different graphical representations of a set of data
Example:Compare same set of data shown on a table, line graph, and circle graph.
Standard ‘as written’
Less Complex More Complex
‘Entry Points’
Essence of standard:Represent information
in different formats
Access to the General Curriculum:Mathematics
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 8
Used to calculate the Performance Level:
Completeness of portfolio
Level of Complexity (difficulty of standards)
Demo of Skills and Concepts (accuracy)
Independence (cues/prompts/assistance)
Plus,
Self-Evaluation (monitor, self-correct, reflect)
Generalization (varied instructional approaches)
Scoring Criteria
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 9
MCAS-Alt Scoring Rubric: Demonstration of Skills and Concepts
How accurate were the student’s responses?
M 1 2 3 4
The portfolio strand contains insufficient information to determine a score.
Student’s performance is primarily inaccurate in this strand.
(0-25% accurate)
Student’s performance is limited and inconsistent with regard to accuracy in this strand.
(26-50% accurate)
Student’s performance is mostly accurate in this strand.
(51-75% accurate)
Student’s performance is accurate and of consistently high quality in this strand.
(76-100% accurate)
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 10
MCAS-Alt Scoring Rubric: Independence
To what degree were prompts used; How independent were the student’s responses?
M 1 2 3 4
The portfolio strand contains insufficient information to determine a score.
Student requires extensive verbal, visual, and physical assistance to demonstrate skills and concepts in this strand. (0-25% independent)
Student requires frequent verbal, visual, and physical assistance to demonstrate skills and concepts in this strand. (26-50% independent)
Student requires some verbal, visual, and physical assistance to demonstrate skills and concepts in this strand.
(51-75% independent)
Student requires minimal verbal, visual, and physical assistance to demonstrate skills and concepts in this strand.
(76-100% independent)
11
Setting Performance Levels
Use score combinations to describe characteristics of student’s performance: Reasoned Judgment
Example:LC=3, Acc=4, Ind=3 shows student’s
performance is primarily accurate and independent, although below expectations for grade level.
Example:LC=3, DSC=2, Ind=2 shows student’s
performance is limited/inconsistent and student requires frequent prompting/assistance.
University of Maryland – Alternate Assessment Conference
October 11-12, 2007
Score Combination Tables
Level of Complexity=3
2007 CCSSO Large Scale Assessment Conference Making a Case for MCAS-Alt Validity
12
University of Maryland – Alternate Assessment Conference
October 11-12, 2007
Level of Complexity=2
Demo of Skills:
Ind
ep
en
den
ce:
1 2 3 4
(0-25%) 1 Aw Aw Aw Aw
(26-50%) 2 Aw Aw Em Em
(51-75%) 3 Aw Em Pg Pg
(76-100%) 4 Aw Em Pg Pg
Demo of Skills:
Score Combination Tables (continued)
Level of Complexity=5
13
University of Maryland – Alternate Assessment Conference
October 11-12, 2007
Level of Complexity=4
Demo of Skills:
Ind
ep
en
den
ce:
Demo of Skills:
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 14
Technical Validity and Reliability: Some Tricky Areas for MCAS-Alt
“Test item inter-relationship” But, tasks are selected and/or designed by teachers, and There is little standardization across portfolios
“Assessment reflects full range of content standards” But non-regulatory guidance says these students won’t
necessarily access all the standards, and Portfolios cannot cover all the standards, only those that were
taught Validate that targeted skills shown in the evidence are
based on grade-level content standards Is an external alignment study necessary?
“Reliability of scores” when responses are so diverse One purpose of MCAS-Alt: Instructional improvement
How to document that this occurred?
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 15
“Did the MCAS-Alt Meet Its Intended Purposes?”
Tell our story: Did the assessment do what we said it would do? If not, how did we fix it?
This criterion allowed us to document… Whether the student was provided access to
curriculum Whether new, challenging skills were taught How well student learned new skills, concepts, content Whether teaching and learning improved as a result of
MCAS-Alt
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 16
Document What Happened: Validating the Development Process
We tried to get the right people at the table We carefully documented all decisions:
Determine purpose(s) of the alternate assessment What we want to measure (scoring rubric) Describing the student’s performance (descriptors) Calculating a score (scoring rules) Translating scores into performance levels
(standard setting) Where one PL ends and another begins (cut scores) Aligning content and validating the alignment Continuous improvements to the system
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 17
Who Contributed to the Validation Process?
Curriculum Framework writers served on panels to develop the Resource Guide to the Frameworks for Students with Disabilities Content specialists defined the “essence” of standards and
“entry points” at various levels of complexity
Special educators pushed them to go lower Diverse stakeholders shared their perspectives Technical advisors helped set performance standards,
using reasoned judgment of each “score combination” Contractors told us what others had tried, and what
might work Scorers linked the portfolio evidence to the required
standard using the Resource Guide, with 94% IRC
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 18
MA Department of Education (781-338-3625) Dan Wiener – [email protected] MCAS-Alt Website:
www.doe.mass.edu/mcas/alt
Resources
MCAS-Alt:The Evolution of a Validity Argument
Charles A. DePascaleNational Center for the Improvement of Educational
Assessment
University of Maryland – Alternate Assessment Conference October 11-12, 2007
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 20
The Evolution of a Validity Argument
Defining the purposes of the assessment Identifying the multiple uses of the
assessment and the populations of students
Specifying the inferences that would be supported by the assessment
Determining that one “set of rules” and procedures would not be sufficient
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 21
The Evolution of a Validity Argument
Designing the system Building checks and balances into
the system Documentation:
– Understanding the extent to which documentation is the system
– Understanding the importance of documentation of the system
University of Maryland – Alternate Assessment Conference
October 11-12, 2007 22
The Evolution of a Validity Argument
Flexibility and Standardization (Gong & Marion, 2006) Making decisions about where to be
flexible and where it is necessary to standardize.
Making adjustments to enhance validity Adopting an continual improvement
approach Determining when and how to make
changes to improve the system.