Power Standards and SMART Goals; Expectations for Assessment and

Data-based Decision-Making.Presented by:

Ben Ditkowsky, Ph.D.

Waukegan Unit School District 60

Power Standards

• A power standard refers to those few things that are really important to teach (Reeves). – BIG IDEAS (Kame’enui & Carnine)– Enduring understandings (Wiggins & McTigue)

• The key question here seems simple:• What is it that we want students to know

and be able to do?

The practical reality of NCLB

If we are going to be able to “leave no child behind” the state standards, then we are going to have to have a way of determining how far behind they are, from where we expect them to be, so that we can do something to accelerate their academic growth.

…a way of determining how far behind they are, from

where we expect them to be

That means we need to use data to make decisions

Uses of data? + -Good Use

Plan and group for instruction

Monitor student progress for a goal

Identify a student as “at –risk”

Identify new goals for students who have exceeded

Promote student reflection and goal setting

To adjust teaching practices

Poor Use

Grade a pretest and send home

To ensure file folders are not empty

Meet district compliance

Lock students into a group

Make high – stakes decision based on one test

To decide who has failed to meet standards (grade retention)

Increase teacher anxiety

Plan and group for instructionPlan and group for instructionGrade a pretest and send homeGrade a pretest and send homeTo ensure file folders are not emptyTo ensure file folders are not emptyMeet district complianceMeet district complianceMonitor student progress for a goalMonitor student progress for a goalIdentify a student as “at –risk”Identify a student as “at –risk”Identify new goals for students who have Identify new goals for students who have exceeded expectationsexceeded expectations

Lock students into a groupLock students into a groupMake high – stakes decision based Make high – stakes decision based on one teston one test

To decide who has failed to meet To decide who has failed to meet standards (grade retention)standards (grade retention)Promote student reflection and goal settingPromote student reflection and goal settingTo adjust teaching practicesTo adjust teaching practicesIncrease teacher anxietyIncrease teacher anxiety

When we use data we need to FOCUS: Things to consider

• Entry level skills– If students come in with less skill than we expect

then we need to get them to learn more with the same amount of time.

• Efficiency and Effectiveness of Instruction– Time on task– Intensity of instruction

• Data-based decision-making– Frequency of Assessment– Use of data to drive instructional decision-making

Entry Level Skills

• Typical students have a benefit over low performing students.• They come in higher, so they don’t need to gain as much• Keep in mind that we need to make sure students have

prerequisite skills

MoreSkill

LessSkill

Instruction

• To make up the difference, instruction should:– Be focused on the goal– Maintain student time on-task– Provide sufficient practice to ensure students have the opportunity to learn

MoreSkill

LessSkill

Assessment

• The more dire the student needs, the more carefully we should watch their progress.

• When students are not progressing, we need to change what they receive in terms of instruction.

• The more dire the student needs, the more carefully we should watch their progress.

• When students are not progressing, we need to change what they receive in terms of instruction.

Monitoring progress requires us to pay attention

• Is the instruction working?• Is it intense enough?• Do we need to change what we are

doing?MoreSkill

LessSkill

Use Assessments from the Curriculum

• To determine when to change instruction

• Rule: 3 data points below the aim-line

MoreSkill

LessSkill

Continuously Evaluate• Is it working now?

• Well enough to meet the goal?

MoreSkill

LessSkill

Continuously Evaluate• Is it working now?

MoreSkill

LessSkill

Set Goals, and make them smart!• Goals should be

Specific and Strategic

• Goals should be Observable and

Measurable

• Goals should be Acceptable

Action oriented and Attainable

• Goals should be Realistic and

Relevant

• Goals should be Trackable and

Time-bound

•

S•

M

•

A•

R

•

T

How do we know what goals to set?

Use Power StandardsPower Standards:

What is it that we want students

to be able to DO?

Curriculum-Based Measurement

• Does how quickly a student can orally read relate to how well they do on ISAT?

• Will a 5th grade student who…– … reads 120 cwpm meet standards? – What about a student who can read 150

cwpm, will they meet?

Think – Pair -Share

1. Does fluency matter? Yes/No

2. How many words should a student in your class be able to read in a minute?

3. Why do you think so?

Does Fluency really matter?

1. Does it matter? Yes/No

2. How many words should a 5th grade student read in a minute?

Let’s consider some data…

• How many correct words did 5th grade students read in one minute?

• Does the number of words that students can read in one minute relate to how well they do on the ISAT?

Kelsey read 120 words

Dorothy read 225 words

Sarah read 147 words

Mark read 175 words

How many words can they read in a minute?

Let’s reorganize things

• Some times we need to change the way we look at things before we can FOCUS.

Setting up a chart to compare scores

Exceeds

Meets

Below

Warning

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10 20 30 40 50 60 70 80 90 100

110

120

130

140

150

160

170

180

190

200

210

220

230

240

250

260

270

280

290

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

Before ISAT testing what we knew was how Before ISAT testing what we knew was how fast students read a grade level passage.fast students read a grade level passage.

Kelsey read 120 words

Dorothy read 225 words

Sarah read 147 words

Mark read 175 words

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10 20 30 40 50 60 70 80 90 100

110

120

130

140

150

160

170

180

190

200

210

220

230

240

250

260

270

280

290

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

Setting up a chart to compare scores

A note about “Scatter – Plots”

Scatter-Plot can help us to see– How scores from two different tests

relate to each other. – Where we should set cut-points for

students meeting expectations – How we can increase student’s

chances to meet ISAT Standards.

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10 20 30 40 50 60 70 80 90 100

110

120

130

140

150

160

170

180

190

200

210

220

230

240

250

260

270

280

290

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

Looking at relations

Let’s see how Dorothy did on ISAT…

Dorothy read 225 Words in a minuteAnd Dorothy obtained a 169 on ISAT

How did she do?

BDitkowsky@Waukeganschools.org

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10 20 30 40 50 60 70 80 90 100

110

120

130

140

150

160

170

180

190

200

210

220

230

240

250

260

270

280

290

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

Let’s consider…

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10

20

30

40

50

60

70

80

90

10

0

11

0

12

0

13

0

14

0

15

0

16

0

17

0

18

0

19

0

20

0

21

0

22

0

23

0

24

0

25

0

26

0

27

0

28

0

29

0

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

100% of (62, 5th grade) students who read less than 120 wordsper minute were below state standards on ISAT

Remember Kelsey?

Students who read less than 120 correct words per minute

She read 120 words correctly in a minuteShe obtained a score of 150 on ISAT

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10

20

30

40

50

60

70

80

90

10

0

11

0

12

0

13

0

14

0

15

0

16

0

17

0

18

0

19

0

20

0

21

0

22

0

23

0

24

0

25

0

26

0

27

0

28

0

29

0

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

100% of (62, 5th grade) students who read less than 130 wordsper minute were below state standards on ISAT

Students who read less than 130 correct words per minute

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10

20

30

40

50

60

70

80

90

10

0

11

0

12

0

13

0

14

0

15

0

16

0

17

0

18

0

19

0

20

0

21

0

22

0

23

0

24

0

25

0

26

0

27

0

28

0

29

0

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

Students who read less than 140 correct words per minute

77% of (62, 5th grade) students who read less than 140 wordsper minute were below state standards on ISAT

What can we infer about students who are not fluent readers and ISAT?

(<150 cwpm)

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10

20

30

40

50

60

70

80

90

10

0

11

0

12

0

13

0

14

0

15

0

16

0

17

0

18

0

19

0

20

0

21

0

22

0

23

0

24

0

25

0

26

0

27

0

28

0

29

0

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

75% of (62, 5th grade) students who read less than 150 wordsper minute were below state standards on ISAT

Think – Pair - Share

What are two reasons why you think the speed at which our students read influences their scores on high – stakes tests?

1.There is time to complete the test.

2. They don’t have to spend time decoding, instead they can spend time comprehending.

Let’s consider just the students who read more than 150 correct words per minute

Grade 5

0

20

40

60

80

100

120

140

160

180

200

220

240

0 10 20 30 40 50 60 70 80 90 100

110

120

130

140

150

160

170

180

190

200

210

220

230

240

250

260

270

280

290

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

90% of (62, 5th grade) students who read more than 150 wordsper minute were above state standards on ISAT.

Remember Mark?

And Dorothy?

Place your bets

• If you were going to bet that a student was going to meet expectations on the ISAT who would you choose?– A student who reads 120 cwpm?

• 100% of our sample did not

– A student who reads 130 cwpm?• 100% of our sample did not

– … 140 cwpm? Or 150 cwpm?• 77% and 75% of our sample did not

– What about a student who reads more than 150 cwpm?• 90% of our sample of students reading more than 150

cwpm met State Standards.

– Pick your student.

For 5th grade students• By April 2, 200x• When given a randomly selected passage from the

5th grade Reading Series and one minute to read• 90% of students will orally read• 150 words correctly with

95 to 98% accuracy

Reading CBM: An Instructional Focus by grade level

• Specific• Measurable• Action-oriented• Relevant• Time-bound

Consider the data:90% of 5th grade students who read 150 words correctly met ISAT standards

Think – Pair - Share

What are two factors that WE CAN CONTROL, other than the speed at which children read, that might influence their scores on high – stakes tests?

1. Vocabulary

2. Comprehension strategies

3. Other Basic skills

4. Language skill

5. Test – taking strategies, etc…

Grade 3

100

120

140

160

180

200

220

R-CBM

ISA

T

Exceeds

Meets

Below

Warning

92% of students who read 94 CWPM or more met ISAT Standards

What about 3rd Grade?

Can you see a pattern?

93% of students who read 115 CWPM or more met ISAT Standards

100% of students who read 140 CWPM met ISAT Standards

9/14 : 64% of students who read less than 94 did not meet ISAT Standards

Should Oral Reading Fluency be a FOCUS for third grade?

Consider this: – 92% of 3rd students in the (2 school sample) who read more

than 115 words correctly in one minute met ISAT standards.

– Fewer than 10% of students who failed to read at a target rate of 85 cwpm met ISAT standards.

• Make goals S, M, A, R, T

By April 2, 200x, when given a randomly selected passage from the 3rd grade Harcourt Reading Series and one minute to read, 90% of 3rd grade students will orally read 94 words correctly with 95 to 98% accuracy

Re-confirmation of data (Grade 3 N > 500)

Curriculum-Based Measurement and High Stakes Testing

120

130

140

150

160

170

180

190

200

210

220

0 20 40 60 80 100

120

140

160

180

200

220

240

260

Reading (CWPM)

Rea

ding

IS

AT

Sca

le S

core

27%

73%

66%

34%

90%

10%

Grade 3 (N =537)

Re-confirmation with data (Grade 5 N > 500)

Curriculum-Based Measurement and High Stakes Testing

120

130

140

150

160

170

180

190

200

210

220

0 20 40 60 80 100

120

140

160

180

200

220

240

260

Reading (CWPM)

Rea

ding

IS

AT

Sca

le S

core

36%

64%

61%

39%

88%

12%

Grade 5 (N =523)

What should we expect?• Some variation is expected in oral reading fluency

depending on passage difficulty, but clear targets are supported:

• Grade 3– Fall - 60 is cause for concern 80 is a target– Winter - 75 is cause for concern 100 is a target– Spring - 90 is cause for concern 115 is a target

• Grade 5– Fall - 95 is cause for concern 130 is a target– Winter - 125 is cause for concern 145 is a target– Spring - 140 is cause for concern 155 is a target

What about Vocabulary?

• A pilot for middle school

Grade level Teams

Identify critical vocabulary

Student friendly definintions

Vocabulary Matching - CBM1 Science

Vocabulary Matching

Directions:

You will have 5 minutes to work on this activity. Read through the definitions below. Beside each definition, write the number of the word that best matches that definition. Remember, not all of the words will be used.

Name: Date: Definitions Word Bank

The planet you live on (sample) 1 density

Anything that has mass and volume 2 respiratory system

Force of attraction between two objects 3 Momentum

In an experiment, the part that is changed on purpose 4 matter

Force needed to change movement 5 manipulated variable

The maintenance of a stable internal environment 6 scientific

The amount of matter in a given space; mass per unit volume 7 accurate

Correct 8 scientific method

All of space 9 immune system

In an experiment, the part that is measured 10 Universe

Force of a moving object 11 responding variable

Way to compare things 12 ecosystem

Study of genetics 13 chemical reaction The process by which one or more substances undergo change to produce one or more different substances 14 Inertia A series of steps that scientists use to answer questions and solve problems 15 atom

Relating to science 16 Gravity 17 Measurement 18 Earth

A collection of organs that gather and interpret information about the body's internal and external environment and respond to that information. 19 adaptation

Living and nonliving organisms interacting 20 homeostasis

Relationship among organisms 21 Interaction

A prediction based on available information. 22 Heredity

23 reproduction

24 Volcano

25 hypothesis

26 nervous system

18

5142017101131722

13

8

6

26

1221

416

25

Administration and Scoring Vocabulary Matching

• VM Curriculum-Based Measurement can be individually or group administered.

Teacher StudentReads standardized directions Reads for two to five minutes.

Monitors student performance on practice items

Monitors students while they take the test.

Writes the number of words matching definitions

Counts the total number of correctly circled items

Does Vocabulary relate to how well students do on the state test?

• How many students who obtained a score of fewer than 5 on a VM (5 minutes) measure met standards for grade 7 science?

• How about 10?

• What about more than 10 correct?

34%

65%

97%

Does Vocabulary relate to how well students do on the state test?

Vocabulary and Science ISAT

0

10

20

30

40

50

60

0 2 4 6 8 10 12 14 16 18 20VM Score

No

. S

tud

ents

Meet

DNM

Likely Likely NOT to MeetNOT to Meet

Likely Likely to Meet Standardsto Meet Standards

The Measurement Pool

• What words will a successful student know by the end of the school year?

• The student is likely to know some of the key words before you begin.

• As the year progresses, more words are taught, and more words are learned

How can writing be assessed at the classroom level

A first grade classroom example, goals and data:

By March 19, 200X, when given a writing prompt, 90% first grade students will write a complete sentence in the language of instruction that directly includes the prompt in the response with an explanation (quality).

7

6

5 38%

4 50%

3 10%

2 1%

1

0 1%5 10 15 20 25 30 35 40 45 50 55 60 65

Time 1

By December 19, 200X, when given a writing prompt, 90% of first grade students will write a complete sentence in the language of instruction.

A complete sentence includes (1) a noun and a verb (2) begins with a capital letter, (3) ends with punctuation, (4) has a space between

words. 5) To exceed means more than one sentence with 4 elements.

88% met the goal

7

6 57%

5 36%

4 7%

3

2 High Quality IndicatorBeginning Quality Indicator

1 Quality absentInsufficient quantity indicators

05 10 15 20 25 30 35 40 45 50 55 60 65

Time 2

By January 19, 200X, when given a writing prompt, 90% of first grade students will write a [part 1] complete sentence in the language of instruction that [part 2]

directly includes the prompt in the response (quality).

A complete sentence includes (1) a noun and a verb (2) begins with a capital letter, (3) ends with punctuation, (4) has a space between words. (5) more

than one sentence with 4 elements,

93% met part 1 of the goal

77% met part 2 of the goal

What do we need to do when not everyone meets?

• Meet with peers / peer mentors (Grade 1 Team meetings)

• Instructional support• Increasing focus and intensity of instruction

By January 19, 200X, when given a writing prompt, 90% of first grade students will write a [part 1] complete sentence in the language of instruction that [part 2] directly includes

the prompt in the response (quality).

93% met part 1 of the goal

77% met part 2 of the goal

By March 19, 200X, when given a writing prompt, 90% of 1st grade students will write a complete sentence in the language of instruction that directly includes the prompt in the response with an explanation (quality).

A complete sentence includes (1) a noun and a verb (2) begins with a capital letter, (3) ends with punctuation, (4) has a space between

words. (5) more than one sentence with 4 elements,

7

6 68%

5 22%

4

3

2 High Quality IndicatorBeginning Quality Indicator

1 Quality absentInsufficient quantity indicators

05 10 20 15 25 30 35 40 45 50 60 65 70

Time 3

78 %

Goal setting at the Grade level

A Kindergarten Center example…

Making Instructional Decisions based on Data

1. Identification of Needs

2. ValidateNeeds

3. Plan Development

4. PlanImplementation

5. Plan Evaluation and

Modification

Is the difference between what is expected and what is occurring?

What does the student need?Do we have enough data to confirm or refute a hypothesis?

What is the goal of instruction?What materials will be used?How will progress be monitored?

How will implementation integrity be ensured?Was the intervention plan successful?

The Plan:Three Tiers of Intervention

• Defined: A data – driven model for differentiated instruction for all students

• Focus on what works– Scientifically Based Reading Instruction– In the context of a self – correcting model of

decision-making

• Identifies / verifies need then organizes support that students need

Three Tiers, with an Emphasis on Prevention at Each Level

• Tier III (FEW students) Tertiary Intervention– Reduce complications, intensity,

severity of current problems

• Tier II (SOME students) Secondary Intervention– Reduce current cases of

academic and behavior problems

• Tier I (ALL students) Primary Intervention– Reduce new cases of academic

and behavior problems

Intensity of Need

Inte

nsi

ty o

f se

rvic

e

Tier 1:Benchmark Instruction(Typical)

Tier 2:Strategic Instruction

Tier 3:Intensive Instruction

Students who need more… receive more

Intensity of Need

Inte

nsi

ty o

f se

rvic

e

Tier 1:Typical Instruction

Tier 2:Strategic Instruction

Tier 3:Intensive Instruction

2004-2005 Meeting Expectations: In Phonological Awareness

By June 1st 200X, when given a DIBELS Phoneme Segmentation Fluency Probe, 95% of students will orally identify more than 10 phonemes per minute (Emerging phonological awareness or better). Sixty-percent (60%) of Kindergarten students will identify 35 or more phonemes per minute (Established Phonological Awareness).

60

50

40

30

20

10

0

Co

rrec

ts

FallBenchmark

Oct Nov Dec Jan WinterBenchmark

Feb Mar

ISF

LNF

PSF

NWF

Intensity of Need

Inte

nsi

ty o

f se

rvic

e

Tier 1:Typical Instruction

Tier 2:Strategic Instruction

Tier 3:Intensive Instruction

Student: Jen• September: Tier 3• October: Strategic Monitoring• Successful intervention confirmed

with progress monitoring• December: Tier 1

Three Tiers in action

Monitoring progressThink graphically

How

muc

h or

how

man

y?

Time

PLEP

Collect enough data to bereasonably confident that you know baseline (e.g., the median of 3)

What is your goal?

What should progress look like?

Collect and graph data!

If data don’t show growth change something!

If data do not match the expected course, change something!

Keep changing until the data look the way you expect them to look!

Leaving No Child Behind• We agree with the goal of No Child Left Behind

– Safe schools– Qualified teachers, etc.– All children reading at grade level

• The practical reality is that while we may not be able to demonstrate on level results on State tests, SMART goals will show us which students need more/different instruction to improve their rate of progress.

• SMART goals will empower teachers and allow them to celebrate their success with students.

BIG IDEAS• Determine your power standards • set smart goals• Monitor progress• If it’s not working, or it’s not working well

enough change it

BIG IDEAS• There are things that we can monitor that can help us see how

we are going to do.– Whether we are talking about student academic achievement,

appropriate behavior, or diffusion of innovations – the first step is identifying specifically what we are going to measure (Power Standards).

• We know that once we have the Basics,– Some need more to “meet” expectations.– And it is always nice to exceed.

• We can be reasonably confident when we answer the question, “How are we doing?” before answering “How did we do?”

…and just for fun, how about a high school example

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

A C T Math Score (2004)

Cum

ula

tive

(es

timat

ed)

prob

abili

ty o

f m

eetin

g

sta

ndar

ds o

n P

S A

EThe relation between A C T and meeting standards on P S A E

19

59%

The odds of meeting

standards on PSAE for a student who scored 19 on ACT is 58.7%

The odds of meeting

standards on PSAE for a student who scored 19 on ACT is 58.7%

A C T Cut scores for Reading and Science

Reading ACT SCORE

Predicted probability

2 0.0%5 0.0%6 0.0%8 0.1%9 0.2%

10 0.4%11 0.8%12 1.7%13 3.4%14 6.7%15 12.9%16 23.3%17 38.4%18 56.2%19 72.5%20 84.4%21 91.7%22 95.8%23 97.9%24 99.0%25 99.5%26 99.8%27 99.9%28 99.9%29 100.0%30 100.0%31 100.0%32 100.0%33 100.0%36 100.0%

Science ACT SCORE

Predicted probability

Mean7 0.0%8 0.0%9 0.0%

10 0.0%11 0.0%12 0.0%13 0.0%14 0.0%15 0.1%16 0.6%17 2.6%18 10.0%19 31.8%20 66.2%21 89.2%22 97.2%23 99.3%24 99.8%25 100.0%26 100.0%27 100.0%28 100.0%30 100.0%32 100.0%33 100.0%

Substantial jumps in probability of meeting standards for Reading and Science (PSAE) were obtained based on ACT as well

Approximately 90% of students who score 21 on any given sub-test meet standards on PSAE

So what does this tell us?FINDINGSFINDINGS

The probability of meeting PSAE standards increase by nearly 30% by increasing from an ACT score of 17 to 18 in Reading; 18 to 19 in Math; and 19 to 20 in Science. Approximately 90% of students obtaining a score of 21 met standards regardless of the subject area.

These findings are reliable - Analysis of a data set representing more than 10,000 11th grade students resulted in nearly identical findings

IMPLICATIONIMPLICATION

Although this is not causal (i.e., 19 does not cause PASE = Meet), the dramatic increase in probability does compel us to investigate this relation.

ACTION QuestionACTION Question

Is there something that teachers can do to increase the chances of students obtaining a score of 18,19,20 (for Reading, Math, Science) on ACT?

FOCUSFOCUS

We can use the practice ACT data to drive the focus of instruction!

On which type of item ought we focus?

• All items were examined and sorted according to difficulty.

• The raw score equating to 21 was determined.• Items were coded as Most difficult, Difficult,

FOCUS and Easiest

SuggestionSuggestion: Use the actual practice test, examine the items as a team and with students to determine why those items were missed. Don’t teach the test, but do teach the concepts.

With which students ought we focus and on what?• All scale scores were computed.

• Scores were compared with probabilities to designate the risk status of students – regarding the likelihood of meeting standards on PSAE – based on data and a little guessing.

• Scores were sorted by class – English, Science, Math, unidentified class and – color-coded.

HR

IR

LR

NR

High Risk

Indeterminate Risk

Low Risk

Negligible Risk

Likely to be on academic warning

May need to assess further, strategize instruction

Maintain current program

Likely to meet PSAE if it were given today

Suggestions

• Use the data to begin to strategize who needs how much of what.

• The more you analyze the actual test the more you will be able to assist your students in being ready for the concepts that are being tested