Finding CoherenceFocus, Measure, & Connect

The Measure Sessions!November, 2014

http://digitallearningforallnow.com

http://www.slideshare.net/jpcostasr

costa@educationconnection.org

Jonathan P. Costa

Student DataDriving

InstructionalPractices &

DecisionMaking

InstructionDriving

ImprovedMeasures

ProfessionalPracticeDriving

ImprovedMeasures

SystemsDriving

ImprovedMeasures

Adult DataDriving

ProfessionalLearning &

DecisionMaking

OrganizationalData

DrivingSystemsDecisionMaking

S-G Goals for Student Learning

•Connecticut Core/21st

Century Skills & Content

A-G Goals for Professional Learning

•Evaluation & support goals, SLOs, focus goals & other

O-G Goals for Building and District

• Improvement targets related to DPI, SPI or other goals

DPI

SPI

5

10Other…

O-M

Building & District

Measures

45

40

(5/10)

Smarter Balanced &

other valued summative, formative,

standardized and non-

standardized measures

S-M

Student Measures

District or building

level plans or

strategies

Job focused

& aligned with

45

5

40

10

CC/21CS goal

aligned teaching methods

& strategies

Domain One

Student Learning

Domain Two

Professional Learning

Domain Three

Organizational Learning

Goals

MissionLeadership

Focus MeasuresPractices

G = Goals P = Practices M= Measures Jonathan P. Costa S= Students A = Adults O = District/Building

S-P

Instructional Practices

A-PProfessional

Learning Practices

O-P

Building & District Practices

A-M

Professional Measures

Coherence Pathways – 5,000 Foot View Leadership

FocusMeasureConnect

Mission

To prepare every student for

learning, life, and work in the 21st

century.

Three Principles of Coherence

Measure what you

value

Value what you measure

Priority Student Learning

Priority Adult

LearningPriority Systems Learning

Student Learning

Adult Learning

Systems Learning

MeasureFocus Connections

S-G Goals for Student Learning

• What are your most important goals for learning?

How will you know if you are improving (Measure)?

Using the identified skills, create an assessment plan that includes; a. existing

standardized data (CAPT/CMT), b. identifies potential Smarter Balanced/Common Core

assessments, and c. creates local assessments that can be used for formative

measurement of student performance of these skills.

S-M

Student Measures

G = Goals P = Practices M= Measures Jonathan P. Costa S= Students A = Adults O = District/Building

Leadership

FocusMeasureConnect

Mission

To prepare every student for

learning, life, and work in the 21st

century.

Coherence Pathways – Essential Connections

Goals

MeasuresPractices

Focus

Coherence Is Supported by Assessment

Jonathan P. Costa

Value

Reliability

What Student Data Do You Have and Use?Skill Set

Critical Attributes Assessments & Data Points

(Current)Assessments & Data Points

(Needed)

Critical Stance

Based on Evidence

M3/E4

Construct and engage in

viable arguments based

on evidence and critique

reasoning of others.

SMP 3 – Distinguish correct logic in

reasoning from that which is flawed.

SMP3 –Build a logical progression of

statements to explore the truth about

conjectures.

SMP 3 -Use stated assumptions,

definitions, and previously established

results in constructing arguments.

SMP 3 - Recognize and use counter

examples.

SMP 3 – Contextualize and

decontextualize quantitative

relationships

E4 - Question an author’s or speaker’s

assumptions and premise

21 - Effectively analyze and evaluate

evidence, arguments, claims and

beliefs

21 - Synthesize and make connections

between information and arguments

21 - Interpret information and draw

conclusions based on the best analysis

Reflections on Assessment… • To be worth anything, all assessment must do one of these two things….

1. Improve student learning (formative)

2. Create both student and teacher accountability (summative)

• How would you define/characterize these two categories of assessment?

– Formative:

• Why –

• How -

– Summative:

• Why –

• How -

Reflections on Assessment…

- Finding the Balance Point -

- Given what we have discussed about the purpose and process of formative and summative assessments… what do you think is the most appropriate and productive % time split between these two assessment purposes (of the total time devoted to assessment – how much for formative and how much for summative)?

-Form / Sum -

- Given what we have discussed about the purpose and process of formative and summative assessments… what do you think is the CURRENT PRACTICE between these two assessment purposes in your district (of the total time devoted to assessment – how much for formative and how much for summative)?

-Form / Sum -

Reflections on Assessment…• To effective, formative or summative assessments should be:

1. Generally Reliable (as opposed to statistically so)

2. Valid (reflects what it is intended to measure)

3. Helps the learner improve his/her performance (formative mostly)

4. Can be tracked and analyzed over time (summative mostly)

• What is your general sense of the state of quality of these elements in your school/district?

Formative Summative

Warm Cool Warm Cool

Reflections on Assessment…

Other data points: Required, by grade, department, standards aligned, type of administration, time required, testing window, vendor, ROI

1. Philosophy – the role that assessment plays in your system – differences of formative and summative – thinking about the differences between growth and accountability and how to balance them in practice.

2. Technical Capacity

• Constructing effective rubrics – measuring what you value

• Designing rubric aligned tasks – connecting it to practice

• Scoring protocols – build capacity to judge student work

• Data collection and analysis tools – how to use the results

3. Systems effectiveness

• Organization of the weight of your assessments – inventory

4. Other?

Building Assessment Capacity

The Biggest Challenge

Getting together and building the structures from which to give students the feedback they need to ensure success and accountability in these priority skill areas is the hardest part of this work.

Please write 5 words that describe last winter’s weather in Connecticut.

Keep this shift in mind…

- From -View Judge Explain

- To -Establish a reliable standard; then…

View Describe Analyze

It’s our natural instinct…Think back to the five words I asked you to

write to DESCRIBE the winter weather in Connecticut…

Pay close attention to the language.

Judgment - Descriptive“reliable” vs. “corroborated by at least

one other source”

“effective” “1st person narrative”

To help this shift…

Something to Think About• There is no shortcut to the amount of time

required to use standards and assessment effectively.

• You either have to define the standard from scratch through descriptive language or you will have to define it through working with your peers to develop anchor sets/models of student work.

• The best strategy is probably a combination of both approaches.

Align Assessments With Goals for Learning

Appropriate

Assessment

Instruments

Recall & recognition

based quizzes, tests,

and activities. Multiple

choice, matching, fill

In the blanks, etc.

(SAT/AP/Exams)

Checklists,

analytic rubrics,

or other agreed upon

skill standards

(AP/Smarter B/Exams)

Holistic and,

analytic rubrics,

or other agreed upon

skill standards

(Portfolios, Demonstrations,

Exhibitions, Etc)

Content(Declarative)

Facts

“Define this term”

Content Skills(Procedural)

Discrete Skills

“Factor this equation”

Meta Skills(Contextual)

Applied Understandings

“Skilled Information Consumer”

Type of

Knowledge

Desired

Appropriate

Assessment

Standards/Scales

Take this quiz, test,

or other appropriate

content focused

assessment instrument.

Answering/responding to

classroom or text based

problems, experiments, and

writing assignments.

Complex projects,

real time explorations,

authentic problem based

skill applications.

Appropriate

Assessment

Tasks

Percentages of

right and wrong

answers on an agreed

upon numeric or letter

grade scale.

1-100/A-B-C-D

Specific criteria tied

to an agreed upon

numeric or

descriptive scale.

1-100/Pass-Fail

Insufficient-Excellent

Specific criteria tied

to an agreed upon

descriptive scale.

Insufficient-Sufficient

Proficient-Excellent

Overlap between columns can and does occur.

Analytic Rubric (Matrix Format)

Clearly defined Tasks and Skills, Some Higher Order Skills

Execute, Complete, Apply, Analyze, Evaluate

Assessing Reading Comprehension

Item Insufficient Sufficient Proficient Excellent

Purpose

Cannot identify the

purpose for most works

Identifies the purpose

for most works

Always identifies the

purpose for the work

Always identifies the

purpose for the work

without hesitation

Inferences

Cannot provide an

inference related to the

piece

Provides an inference

with at least one piece of

evidence to justify

his/her choice

Provides an inference

with more than one

piece of evidence to

justify his/her choice

Provides an inference

with multiple pieces of

evidence to justify

his/her choice

Point of View

Cannot identify the

point of view

Identifies the point of

view and, with

assistance, can give at

least one example

Identifies the point of

view and independently

gives at least one

example with details

Identifies the point of

view and independently

gives more than one

example with details

Opinion

Cannot state agreement

or disagreement with the

point of view or provide

even one non-disputable

fact as evidence to

defend their position

States either agreement

or disagreement with

the point of view and

provides at least one

non-disputable fact as

evidence to defend their

position

Definitively states either

agreement or

disagreement with the

point of view and

provides more than one

non-disputable fact as

evidence to defend their

position

Definitively states either

agreement or

disagreement with the

point of view and

provides several non-

disputable facts as

evidence to defend their

position

Traditional Content Assessments

Factual Recall and Recognition

Activities – Usually based on a

% of correct answers.

Multiple Choice, True/False,

Matching, Fill in the Blank,

Short Answer, Etc.

PowerPoint Skills Checklist

PowerPoint Skills What? I think

I can.

Got it!

1. Use Help – program based

2. Use the Auto-Content Wizard

3. Apply a presentation template

4. Use all 5 slide views and functions

5. Use a variety of slide layouts

6. Change toolbar views

7. Insert clip art, pictures, and/or objects

8. Create and insert a Word Art picture

9. Change your color scheme

10. Change font styles, size, and color

11. Change bullet and line format and color

12. Use an Action Button or HyperLink

13. Run a slide show

14. Change slide transitions

15. Use custom animations

16. Use pre-set animations

ChecklistContent and Simple Tasks

Identify, Recognize,

Recall, Locate

Project Assessment

ProjectCombinations of

All Three

Item Points

Available

Points

Earned

Comments

Brainstorming 5

Contributions from all 2 (g)

Respect for other ideas 3

Discuss and Decide 15

Support each others ideas 10

Use of a voting tool 5 (g)

Draft 25(g)

Shared draft responsibility 5

Neatness and Legibility 5

Draft outline or Web 5

Evidence of feedback 5

Evidence of changes 5

Performance 15

Equality of participation 10

Time limits 5 (g)

Skills Assessment (Rubrics) 40

Social Responsibilities 20

Respond and reflect 20 (g)

Affective Self-Reflection Framework

Skill/Indicator Examples Observations/Evidence Comments/Reflections

Respect:

1.respects self

2.respects and accepts all

members of the community

3.respects property and the school

environment

4.respects others personal space

5.demonstrates empathy

Confident in their own abilities and

appearance

No insults or prejudicial remarks

Use polite greetings and phrases

Never abuses or damages property

Keeping equipment, books, school

related items in good condition

Respects others wishes regarding

personal space and physical contact

Does not engage in abusive behavior

toward self or others

Cares and supports others in need

Expresses concern for others

Responsibility:

1.follows through on personal

commitments or obligations

2.accepts responsibility for

personal actions and behavior

3.demonstrates social and

academic responsibility

Shows up on time for assigned duties

Completes work or assignments in

timely fashion

Does not blame others for personal

performances

Accepts responsibility and restitution

Admits mistakes

Dependable member of a team or

group

© JPC Sr.

2007

A Process for Building Performance Assessments

1. Identify what you are assessing for.

2. Determine critical attributes.

3. Select an appropriate assessment instrument/scale to judge the work.

4. Describe each attribute across the selected scale.

5. Provide models of work across the scale.

6. Assess, and/or have the student assess, the work by applying the ratings and providing explanations.

© JPC Sr. 2007

Editorial Guidelines1. Pick a voice and stick with it. The learner will, the student is able, there is no, etc.

2. Pick a style and stick with it. Lists, narratives, scales, etc.

3. Remember the scale and stick with it. A “1” should be unacceptable, no credit - a “2” should be something that everyone should be able to attain (95%), a “3” should be above average, and a “4” should be very high performing.

4. Commit to the Pareto Principle. Focus on what is most important at every level. Do not be afraid to do a few things well. Remember, if you can’t use these in a classroom environment, they don’t help anyone.

5. Follow your strands through the scale. If one of your critical attributes relates to a frequency issue (number of errors) describe that all the way through the scale. Likewise, if the attribute is a quality issue (clarity of speech) describe that through the scale as well - do not mix and drop.

6. Whenever possible, use descriptive words over judgement words. “Makes multiple spelling errors in every sentence” vs. “Does not use proper English.”

7. Remember the big picture. These tools need to provide us with reliable feedback on student performance. It should be assumed that all students will be asked to participate (with the normal appropriate modifications) and that in order to be considered a “3” a student must satisfy all of the attributes of a “3.” If I have one attribute of a “3” and the rest are at the level of a “2,” I am still a “2.”