© Regents of University of California 1
Functional Validity: Extending the Utility of State AssessmentsEva L. Baker, Li Cai, Kilchan Choi, Ayesha Madni
UCLA/CRESST
Comparing Expectations for Validity Models and for New Assessments: Goals, Approaches, Feasibility, and Impact
Council of Chief State School Officers (CCSSO)
2015 National Conference on Student Assessment
San Diego, California – June 24, 2015
© Regents of University of California 2
So What’s New?Opting Out
• Salient target• Evidence of
benefit • Displaced anger• Transparency
© Regents of University of California 3
Transparency: Expectations Clear and Sensible?
• Better test transparency and utility for public• Specificity in the right places• Support student learning and persistence
© Regents of University of California 4
Today: Feature Analysis
• Argue that tests for “summative” purposes can contribute to transparency of findings to improve learning
• By conducting qualitative and quantitative analyses of tests (and interventions) the veil of obscurity—and what to teach—can be lifted
• FA key element of data-mining
© Regents of University of California 5
Features for Analysis and Design of Assessments
• Identify features of items and tasks on assessments and interventions that may inform teaching, learning, and performance within and across content requirements and grade levels
• Report to improve teaching • Use to design and revise items and tasks
© Regents of University of California 6
How It Works
• Rate components of items/tasks• Low inference features• Features recombined in tasks, items
(game levels, episodes) • Meta-tagged in data• Performance summaries across individual
or clusters of features• Criteria: Significant difficulty, growth, or
complexity
© Regents of University of California 7
Sample CRESST Features: Content, Cognition, Task, Linguistics
• Knowledge—mapped to standards and prerequisites– Content—topics, memory, concepts,
procedures, systems– Representations
• Cognitive requirements and skills– Problem solving components– Communication, inferencing– Pattern detection, situation awareness
© Regents of University of California 8
Task Features
• Surface requirements– Format– Stimulus content, prompts, resources, representations– Game mechanic or interaction engine– Affordances, accessibility, accommodations – Team work requirements – Narrative or scenario content and structure
• Response Requirements– Answer formats– Criteria or scoring rules– Actions or number of types and steps in a response– Essay elements or particular demands
© Regents of University of California 9
Linguistic Features
• Discourse– Complexity or number of ideas in passage or directions– Length– Literal or inferential comprehension– Academic structure, domain-dependent or independent
• Syntax– Sentence patterns, type and variation– Sentence length– Context cues
• Word choice– Academic vocabulary-specific domain– Academic language, type, density
© Regents of University of California 10
Problem Solving Constraints
Single: Increase Vector’s speed to reach stars by reducing amount of friction.
Multiple: Increase Vector’s speed to reach stars but not too fast to avoid hitting dynamite.
© Regents of University of California 11
State Assessment Study - 1
Purpose:• To predict performance from three years of
standards use and attribute results– Rated features of content, cognition,
linguistics, and tasks with high consistency– Tagged every test item in English Language
Arts (ELA) and math for grades 3 & 4 and 7 & 8 for years 2011, 2012, and 2013 by feature
– Features accounted for on average 50% of variance on item difficulty
© Regents of University of California 12
Assessment Study - 2
• Math Grades 4, 8, 11• Previous features augmented by
results of student think-alouds • A total of 70 features identified and
tagged on a sample of math items
© Regents of University of California 13
Assessment Study - 2 Findings
• 4th grade: 16 features were significantly related to difficulty, 10 harder, 6 easier
• 8th grade: 10 features significantly related to difficulty, 6 harder, 4 easier
• 11th grade: 12 features related to difficulty, 7 harder, 5 easier
© Regents of University of California 14
Feature Relationships
• Features across grades– Cognitive load– Representation type– Constructed or multiple responses– Guidance– Linguistics
• Features across ELA and math – Linguistics amount
• Able to predict item difficulty by features
© Regents of University of California 15
Current R&D
• Continuing FA of state level assessments, refining definitions, protocols, and training
• Sub-group and feature interactions• FA of interventions—PBS learning games and videos,
classroom instructional assignments• Linking features of interventions and assessments to
predict performance• Developing two ways of automated feature
extraction • Designing assessments and games using features • Engaging in FA validity studies across projects• Looking for partners
© Regents of University of California 16
Summary
• Feature analysis may make “summative” results useful for improvement
• Multiple purposes for tests• Development implications for tests
and for designing and predicting effects of interventions
Copyright © 2014 The Regents of the University of California. Do Not Distribute
Eva L. Baker
© Regents of University of California 18Back up slides
Back Up Slides
© Regents of University of California 19
State Assessment Functional Validity
• Data to determine year-to-year cohort performance changes – instructional sensitivity?
• Summarized across specified features significantly related to high and low difficulty
• Resulting feature sets accounted on average for 50% of variation of performance
• If confirmed by instructional studies, findings may guide teachers and professional development to improve test performance using invariant
• Guide procurement for re-designed specifications
Note: Cai, Baker, Choi, Buschang, 2014; Baker, Cai, Choi, 2014; Choi, Madni, 2015
© Regents of University of California 20
How It Is Done – Feature Parsing
• Elements are defined and rated which comprise test items and tasks or learning requirements, e.g., linguistics, content elements, detailed cognitive processes
• Each item is re-rated by pairs of trained staff for each feature.
• More granular and operational level of analysis than many currently used approaches
• Features tagged to items in data
© Regents of University of California 21
Purposes of Assessments
• Beyond accountability• Policy linking accountability and improvement • Accountability analyses interfere with guidance
supporting teaching and learning • Can improvement of learning become a useful
function of large-scale tests?• Feature analysis of item and test properties can
yield useful instructional information
© Regents of University of California 22
Mapping Features: Ontologies: Networks of Relationships
SEL
Problem Solving
Content