More than Tests?: ExpandingMore than Tests?: Expandingdesign from tests to instructiondesign from tests to instruction

and learningand learning

Brian GongBrian GongCenter for AssessmentCenter for Assessment

Discussion of “Creating Tests that Measure WellDiscussion of “Creating Tests that Measure Well andand thatthatModel Good Instructional and Learning PracticeModel Good Instructional and Learning Practice” Session by” Session by

Randy Bennett and Edys QuellmalzRandy Bennett and Edys Quellmalz

CCSSO National Conference on Student AssessmentCCSSO National Conference on Student AssessmentMinneapolis, June 28, 2012Minneapolis, June 28, 2012

Overview of commentsOverview of comments

Two really strong projects of national standingTwo really strong projects of national standing

Comprehensive assessment systemsComprehensive assessment systems

Design lessonsDesign lessons Design lessonsDesign lessons

Process lessonsProcess lessons

2Gong – CCSSO - Tests that Measure & Model - 6/28/12

1. Contribution to Comprehensive1. Contribution to ComprehensiveAssessment SystemsAssessment Systems

One assessment taxonomy: formative, interim,One assessment taxonomy: formative, interim,summativesummative

Within interim, another taxonomy of purposes:Within interim, another taxonomy of purposes: Within interim, another taxonomy of purposes:Within interim, another taxonomy of purposes:instructional, evaluative, predictiveinstructional, evaluative, predictive (Perie, Marion,(Perie, Marion, &&

Gong, 2009,Gong, 2009, EMIPEMIP))

A different view: more contextualizedA different view: more contextualizedassessments; relation to curriculumassessments; relation to curriculum

3Gong – CCSSO - Tests that Measure & Model - 6/28/12

Assessment Relation toAssessment Relation toCurriculum/InstructionCurriculum/Instruction

More decontextualized More contextualized

Targeted to standards,generalized away from particularinstructional means to theoutcome (“ends”)

Targeted to standards withincontext of particular instruction(“means” = context)

outcome (“ends”)

Claim & interpretation typicallyfocused on “construct”

Claim & interpretation often interms of construct & content

Comparability & reliabilityvalued, leading to high degree ofstandardization

Some flexibility built in, alwaysin instruction, sometimes inelements of assessment (e.g.,administration)

4Gong – CCSSO - Tests that Measure & Model - 6/28/12

Assessment Relation toAssessment Relation toCurriculum/InstructionCurriculum/Instruction –– 22

More decontextualized More contextualized

Traditional psychometric quality,generalizability, credibility, andlow intrusion valued overtimeliness and detail

Balance shifted towardscontextual detail, timeliness, andlocalized control; try to maintaintraditional psychometric qualitytimeliness and detail traditional psychometric qualitybut not as generalized (?)

Main means of performanceinterpretation linked to constructand scale, and strongmeasurement models

Main means of performanceinterpretation in terms of (multi-dimensional) content/skills,learning theory, and curriculum

Assessment information use nottied to particular instruction

Assessment information used toinform particular instruction

5Gong – CCSSO - Tests that Measure & Model - 6/28/12

Degree of Contextualization,Degree of Contextualization,Amount of Curriculum SpanAmount of Curriculum Span

Curriculum Span - Large

-Sp

arse

Muc

h

Performance assessmentsoften are morecontextualized; assess

6Gong – CCSSO - Tests that Measure & Model - 6/28/12

Curriculum Span - Small

Co

nte

xtua

lizat

ion

-

Co

nte

xtua

lizat

ion

-

Typical large-scale

assessmentCollection of items

Individual item

contextualized; assessaspects of constructs notaddressed well in otherassessments; and oftenhave greater depth but lesscurricular breadth

2. Design Lessons2. Design Lessons

1. Good models of assessment1. Good models of assessment

2. Good models of learning2. Good models of learning

3. Good models of instruction3. Good models of instruction

4. Good models4. Good models of how these relateof how these relate 4. Good models4. Good models of how these relateof how these relate

More than ECDMore than ECD (evidence(evidence--centered design) whichcentered design) whichhas focused on assessment designhas focused on assessment design

7Gong – CCSSO - Tests that Measure & Model - 6/28/12

Design lessonsDesign lessons –– 22

Good modelsGood models of assessmentof assessment

MeaningfulMeaningful –– content, reportingcontent, reporting

UsefulUseful –– clear, actionable, timely, in theory of actionclear, actionable, timely, in theory of action

CredibleCredible CredibleCredible

BoundedBounded –– limitations acknowledged, e.g., areas still beinglimitations acknowledged, e.g., areas still beingworked onworked on

Spanning credibility for traditionalSpanning credibility for traditional psychometrics and thispsychometrics and thisinstructional/learning contextinstructional/learning context

8Gong – CCSSO - Tests that Measure & Model - 6/28/12

Design lessonsDesign lessons -- 33

Good models of learningGood models of learning

Learning progressions (stages of conceptualLearning progressions (stages of conceptualdevelopment, or development of conceptualdevelopment, or development of conceptualrepresentations)representations)representations)representations)

LearningLearning process (e.g., how conceptualprocess (e.g., how conceptualrepresentation andrepresentation and expertise develop)expertise develop)

Type ofType of scaffoldedscaffolded experience (Ramp in, consolidate out)experience (Ramp in, consolidate out)

Role ofRole of feedback, selffeedback, self--assessment, reflectionassessment, reflection

Role of practice, generalizationRole of practice, generalization

Deal with errors or less developed representationsDeal with errors or less developed representations

Etc.Etc.9Gong – CCSSO - Tests that Measure & Model - 6/28/12

Design lessonsDesign lessons –– 33 Good models of curriculum and instructionGood models of curriculum and instruction

Organized experiences to help learner developOrganized experiences to help learner developknowledge/skillknowledge/skill representations/expertise from whererepresentations/expertise from wherelearner is to goal statelearner is to goal state

E.g., Learn with feedback, perform with feedback, reflectE.g., Learn with feedback, perform with feedback, reflect E.g., Learn with feedback, perform with feedback, reflectE.g., Learn with feedback, perform with feedback, reflect

Role of learnerRole of learner (and rest of system, e.g., teach,(and rest of system, e.g., teach, help functions)help functions)

Decision, implementation of quality learning/teachingDecision, implementation of quality learning/teachingexperiencesexperiences (e.g., model of coaching, uses of technology)(e.g., model of coaching, uses of technology)

Good models of how assessment, learning,Good models of how assessment, learning,instructions relateinstructions relate

10Gong – CCSSO - Tests that Measure & Model - 6/28/12

3. Process lessons3. Process lessons

1. “Theory building”1. “Theory building”

Have a theory,Have a theory, model, framework, hypothesismodel, framework, hypothesis

Develop and refineDevelop and refine

Systematically and purposefully gather and use evidenceSystematically and purposefully gather and use evidence

Use the theory and experience (things) to improve eachUse the theory and experience (things) to improve eachotherother

2. Have a committed team2. Have a committed team

3. Use a theory of action about development and3. Use a theory of action about development anduseuse

Be part of the communityBe part of the community11Gong – CCSSO - Tests that Measure & Model - 6/28/12

RecapRecap

Two very good examplesTwo very good examples

Comprehensive assessment systemsComprehensive assessment systems

Design lessonsDesign lessons

Process lessonsProcess lessons Process lessonsProcess lessons

12Gong – CCSSO - Tests that Measure & Model - 6/28/12

For more information:For more information:

Center for AssessmentCenter for Assessment

www.nciea.orgwww.nciea.org

Brian GongBrian Gong

bgong@nciea.orgbgong@nciea.org

13Gong – CCSSO - Tests that Measure & Model - 6/28/12