Planning for the New Measurable Objectives: 2 Critical District Issues with Examples & Background to Help You Address Them

KASB, Topeka, 4 April 2013

Kelly Spurgeon, KSDETony Moss, KSDE

What we hope to cover today:

1. How does the new federal acct. system compare with the old?

2. How can districts best prepare for these changes?

1. An integrated district Measured Objective strategy;

2. Communication strategies for main constituency groups

3. Why do districts need an integrated MO strategy?

4. Nuts & bolts of the Measured Objectives.

Leaving the AYP Mentality:

What we had:• An overemphasis on a state

assessments and a single measure, percent proficient

• Identifying “bubble kids” and getting them over the standards line

• Sometimes teaching to identified test items, not concepts

• Counting the same kids more than once across multiple subgroups

What we will have:• 4 Measured Objectives from

assessments; 5 areas in the new accreditation model, each counting 20%

• Incentives to move every student to their highest possible proficiency level

• Constructed response items that will require deeper conceptual mastery

• One gap measure of the lowest 30%, no duplicate counting

• Overall, an emphasis on improved instruction and instructional support

The big differences:

• A much more nuanced set of measures (good schools can have different profiles)

• College and Career standards—”raise the bar”—so that students are graduating ready for college or a career.

• Why? So they can better compete in a very competitive world.

2 Critical District-Level Tasks:

1. Designing an integrated strategy for making the Measured Objectives; and

2. Developing a communication strategy for each of your 3 key constituencies:

1. Your staff

2. Your board and political stakeholders

3. Parents, the public, and the press.

Why do districts need a Measured Objectives strategy to coordinate school strategies?

• Children’s intellectual and social-emotional development is integrated and hierarchical: each stage depends on the stage that went before;

• Only districts can put all the institutional segments together, from early childhood education and care, through primary, middle, and high school, and keep them all directed toward the same integrated goals.

For example, consider the holes in children’s academic trajectories when measured only by state assessments:

All the other necessary parts—early childhood, student engagement, relationships, school climate and culture, professional development—are missing.

• Collaboration with local pediatricians, early education and child care providers, and social services to improve child development and child-rearing skills before Kindergarten;

• In primary school A, a focus on immediate teacher recognition of any student’s failure to master any competence, including behaviors, and immediate one-on-one work to move the child to competence;

• In middle school B, a focus on weekly career and academic coaching, Individual Plans of Study, and career shadowing;

• In high school C, a focus on improved social climate, social inclusion, continued Individual Plans of Study, project-based instruction, pathway completion, and higher API scores.

• District investments in high-quality, focused, 50-hour PD training.

An Overly Simple Example of a District’s Integrated MO Strategy:

Ages 0 – 5

5 to 9

10 to 14

15 to 18

Let’s now consider districts’ communication needs:

A common question you can expect:

Why all the changes? • To better prepare students for college or a

career, the bar is being raised.• AYP was obsolete and had to be replaced.

In 2015, with the higher standards in the new Kansas Career and College Ready assessments, we can expect for scores to go down substantially.

We have to explain this so it isn’t mis-interpreted.

20102012

20142016

0

100

200

300

400

500

600

700

800

900

1000

reading APImath API

New assessment introduced

Points that Might Be Included in a Staff Communication Plan:

We are in a period of rapid change in assessments and accountability:

o ESEA Waiver replacing AYPo Career pathways assessments comingo School readiness measures comingo A broader set of accreditation measures (state assessments

results only 20%) (rigor, relevance, responsive culture, relationships 20% each)

o KEEP (Kansas Educator Evaluation Project)o Blended assessment in 2014 & Kansas College and Career Ready

assessments in 2015

Staff Communication (cont.)

• The Waiver’s incentives are different from AYP’s—they reward moving every student to the highest proficiency possible;

• Constructed response items will require greater student mastery of concepts; specific test items won’t be identified.

• Constructed response items will have to be checked by staff. Educators will have a view of student proficiency across the grades when they check these items.

• Many MO’s will be re-set in 2015 & 2016 because the assessments will be new.

Points that Might Be Included in a Board or Parent Communication Plan:

• The new assessments are raising the bar, so we should expect that scores will go down.

• This decline is a result of the new assessments, not indicative of worse performance by students or teachers.

• Assessments are designed so that all students miss some items. Some assessments are more difficult than others.

• Different assessments have different plateaus or ceilings.

Example: NAEP Scores

For 17-year olds, NAEP reading scores have only varied 5 points in 42 years.

Now the details of the new MOs:

Academic Performance Index

performance level points per level

# of students total points

exemplary 1,000 262 262,000

exceeds standard 750 268 201,000

meets standard 500 268 134,000

approaching standard 250 60 15,000

academic warning 0 42 0

totals 900 612,000

Assessment Performance Index (API) = 612,000 ÷ 900 = 680

How is the API calculated?

Getting your API data from the Performance by Grade Report:

Let’s compare 2 years of Math API:

level points 2011 # of students

2012 # of students

within category% change

exemplary 1,000 237 262 +10.5

exceeds standard 750 233 268 + 15%

meets standard 500 276 268 -2.9%

approaching standard 250 90 60 -33.3%

academic warning 0 51 42 -17.6%

Result: API went from 645 to 680

Why did Kansas need a new academic performance measure?

• Not many students left below proficient .• The API rewards schools for moving all

students to higher proficiency.• It is more accurate that percent proficient.• It acknowledges the ceiling or plateau—gets

us away from 100% proficient idea.• We could use historic rates of improvement

to set realistic MO goals of improvement.

To differentiate between schools, we looked at the distribution of all schools’ API scores.

Then we divided the distribution into quarters.

Making Progress, Not Making Progress,

Title I Schools

10%Reward Schools

10%Focus Schools

5%PrioritySchools

Title (federal) & Non-Title Categories

4

3

2

1

Reading AMOs are like the Standard of Excellence:

School Category API Range Expected Rate of Improvement / AMO

Cap on % Below Standard

Modeling (Level 4)top 25 percent

API > or = 757

For schools below the 90th percentile, a mean advance of 2 points per year. Above the 90th percentile, whatever improvement is possible.

< or = 5 percent; if not, next lower level

Transitioning (Level 3)3rd quarter

API > or = 703 but < 757

An average yearly advance of 5 points per year

> 5 but < or = 10 percent; if not, next lower level

Implementing (Level 2)

2nd quarter

API > OR = 635 but < 703

An average yearly advance of 10 points per year

> 10 but < or = 15 percent; if not, next lower level

High-Need (Level 1)lowest 25 percent

API < 635

Increments sufficient to enter level 2 or a yearly mean API advance of 15 pts., whichever is greater.

Any school with > 15 percent of its students below proficient is a level 1 school.

Student Growth Percentiles imitate pediatricians’ growth charts.

Normed, percentile bands from the 5th to 95th

Girls’ Length and Weight by Age

Kansas Growth AMO:a relative measure

There are no consequences for not making a building’s growth measure.

Advantages of the Student Growth Percentile Model

SGPs set realistic yearly goals based on each student’s academic peers—students with similar score histories

SGPs map a student’s progress relative to all assessed students

How will growth MOs be displayed?

• How are 270 of my 900 students doing?

• Based on performance categories, not individuals

• No subgroups• See the Performance by

Grade Report• If the LP30 make an API

of 500, also made the gap MO

the Gap: How are the lowest performing 30 % doing?

How is the Gap calculated?

performance level points per level

# of students total points

exemplary 1,000 262 262,000

exceeds standard 750 268 201,000

meets standard 500 268 134,000

approaching standard 250 60 15,000

academic warning 0 42 0

totals 900 612,000

Assessment Performance Index (API) = 612,000 ÷ 900 = 680

900 * 0.30 = 270

performance level points per level

# of students total points

exemplary

exceeds standard

meets standard 500 168 84,000

approaching standard 250 60 15,000

academic warning 0 42 0

totals 270 99,000

API of lowest 30% = 99,000 ÷ 270 = 367

Reducing the Non-Proficient

Reduction of Non-Proficient AMOs

The goal: cut the percentage of non-proficient students (RNP) in half by 2017-2018

Custom RNPs at the district, building, and subgroup levels

Traditional subgroups with an n ≥ 30 will have an AMO determination

No monitored students in the SWDs or ELLs Only the All Students group is merged if less

than 30 (this is still under consideration)

Graduation AMOs

Participation AMOs

Resources

Video clips and fact sheets for Achievement, Growth, Reduction of Non-Proficient and Gap AMOs can be found on the ESEA flexibility waiver webpage: http://www.ksde.org/Default.aspx?tabid=5075

This page can also be accessed by clicking this logo on the KSDE home page

Questions?

Contact the Measurable Objectives Help Desk:Phone: (785) 296-2261Email: mo@ksde.org

Slides that deal with subtopics follow:

The first big hole—early childhood—is where the largest unrealized gains are.

The success of the whole educational enterprise is even more dependent on early childhood than was previously known:

• Working memory and self controls—the ability to delay gratification and control impulses

• Students’ social predilections—empathy and considering others• Students’ behaviors (persistence, engagement, motivation)• Language skills• Some disabling conditions—antisocial behaviors, ADHD—

All have their origins in early interactions and environments.

Some Implications for Districts:

• Large improvements in K-12 education depend on overcoming fragmented services in early childhood and improvements in the quality of family environments.

• The social intelligence and responsiveness of teachers are important--sometimes as important as subject expertise.

• An overemphasis on academics and test scores, to the exclusion of developmentally important events, e.g. fantasy play in the early school years, and social integration, can be developmentally damaging.

Another consequence of ignoring early childhood: large inefficiencies are built into later district and post-secondary results:

returnper $1invested

Why did Kansas need a new academic performance measure?

Relatively few students are available for moving over the proficiency line.

Is the API more accurate than the Percent Proficient?

School T has 91% at Standard or Above, but 75% in the top 2 categories, Exceeds & Exemplary.

School P has 92% at Standard or above, but only 46% in the top 2 categories.

We step away from AYP’s 100% above standard and introduce the concept of a ceiling.

20002002

20042006

20082010

20120

100

200

300

400

500

600

700

800

900

1000

473 477 481525 555

587

618651 661

685 687 701 694

438474 485

523

580617

568621 622

655 653 674 677

reading APImath API

API Average Building Scores, All Students Group, All Public Schools, 2000 to 2012

What is a realistic rate of improvement?

Rates of improvement are larger at the beginning of a new testing cycle. They start to plateau when all of the variables in the system—alignment with standards, student and teacher skills, engagement—begin to reach their limits.

2 pts. / yr

5 pts. / yr

10 pts. / yr

15 or more

19 pts. / yr

Mathematics AMOs

School Category API Range Expected Rate of Improvement / AMO

Cap on % Below Standard

Modeling (Level 4)top 25 percent

API > or = 744

For level 4 schools below the 90th percentile, a mean advance of 2 pts. per year. Above the 90th percentile, whatever improvement is possible.

< or = 6 percent; if not, next lower level

Transitioning (Level 3)

3rd quarter

API > or = 679 but < 744

An average yearly advance of 7 points per year

> 6 but < or = 13 percent; if not, next lower level

Implementing (Level 2)

2nd quarter

API > or = 596 but < 679

An average yearly advance of 13 points per year

> 13 but < or = 19 percent; if not, next lower level

High-Need (Level 1)lowest 25 percent

API < 596

Increments sufficient to enter level 2 or a yearly mean API advance of 15 pts., whichever is greater.

Any school with > 19 percent of its students below proficient is a level 1 school.

How can our measures be made more useful?

• Please tell us what data and charts would help you communicate better with:o Your boardo Your parents o The local press ando Your teachers and staff.

• We need your feedback and suggestions

• To prompt your thinking, some examples follow

Possible Improvements within Current Assessment Reports:

• Population trends • Cohort views of the API & Growth• Grade patterns across years• Adding comparison groups of “schools like

mine” or “students like mine”• Bar graphs that show improvements across

performance levels

How about growth trends?

Will there be comparative growth measures?

What measures would better support instructional improvement?

Emotional Supportclassroom climateteacher sensitivityregard for child perspectives

Classroom Organizationbehavior management / guidanceproductivitymaximizing engagement

Instructional Supportconcept developmentquality of feedbacklanguage modeling

Classroom Assessment Scoring System, Prof. Robert Pianta, et al

What would be meaningful measures to guide reducing the gap?

• Improved early education and child care measures

• Standardized school climate measures (staff collaboration; student trust of teachers)

• Better tools for diagnostic assessments (emphasis on formative assessments)

• Tools to smoothly monitor the effectiveness of interventions