+Utah

Comprehensive

Accountability System (UCAS)

1

Hal Sanderson, Ph.D.Research and Assessment

August 21, 2013

2013

2

+

UCAS performance data is embargoed. September 3rd is the

public release.

3+ Utah Comprehensive Accountability System

(UCAS) Incorporates both student achievement and growth toward improvement in a composite score for each school (range 0 to 600). This is largely based on the Core CRTs.

UCAS provides summary data for the entire school, as well as disaggregated results by ethnicity, and for economically disadvantaged, English language learners, and students with disabilities.

UCAS achievement and growth results are for students who are in school for a full academic year (160 days in membership).

4+Key Features of UCAS Each school can earn up to 300 points in

Achievement and 300 points in Growth. Achievement (Percent of students proficient on

CRTs in Math, Language Arts, Science and DWA). This is similar to previous accountability reports.

Growth (Year-to-year scale score gain compared to like scoring “academic” peers across the state). This is a very different approach to calculating growth/progress.

Subgroup(s). Identifies below proficient students as a single subgroup. Defined as all students who scored below proficiency (level 1 or 2) on the previous year’s CRT

5+Key Features of UCAS continued

Participation. A school must meet the 95% participation rate for the whole school and non-proficient subgroup of 40 students or more in each content area

High School (Graduation Rate = College Readiness)

AMO Information for schools is reported.

6+ ELEMENTARY/MIDDLE SCHOOL UCAS REPORT

Total Points

School Info. Achieveme

nt Points

Growth Points

Report Info

7+ HIGH SCHOOL UCAS REPORT

Total Points

School Info. Achievement

Points

Growth Points

Report Info

Grad Rate Points

8+ Comparison to AYP/U-PASSAchievement

UCAS AYP U-PASSAchievementFor Students who are

Full Academic Year (160 days)

Percent of Students Proficient

Percent of Students Proficient

Percent of Students Proficient

CRTs Included: Language Arts

Yes Yes Yes

Mathematics

Yes Yes Yes

Science Yes No YesDWA Yes No YesParticipation (n => 40)

Yes Yes YesAchievement – is calculated by dividing the number of students scoring proficient or above (Levels 3 and 4) in each content area using the CRT/DWA/UAA tests by the number of students who were enrolled at the same school for the full academic year and took each test.

9+ Comparison to AYP/U-PASS

GrowthUCAS AYP U-PASS

Growth/Progress

For Students who are Full Academic Year

(160 days)

CRT scale score gain compared to like scoring

peers

Safe Harbor (gain in percent

of student proficient)

Progress Score (gain in U-PASS

proficiency)

The Growth or Progress

Question?

Student: How does my scale score

gain compare to students who had

my same prior year(s) score?

What is my percentile rank?

School: Is a greater percent of students proficient than the previous

year?

Student: Has my proficiency level

increased from the prior year (e.g.,

Level 1b to Level 2b)?

UCAS Student Growth Percentile (SGP) – For students taking the CRTs, growth is determined by comparing the performance of a student with all other students in the state with the same past performance (1-3 years of CRT scores). Next, determine how performance in the current year compares with that of the student’s peer group to produce a growth percentile. Range from 0 to 100.

+ 10

2013 UCAS Assessments• CRT LANGUAGE ARTS – Elementary

Grades 3 – 6; Secondary Grades 7 - 11

• CRT MATH - Elementary Grades 3 – 6; Math 7 CRT, Math 8 CRT, Algebra 1, Geometry, Algebra 2

• CRT SCIENCE – Elementary Grades 4 – 6; Secondary 7th Integrated Science, 8th Integrated Science, Earth Systems Science, Biology, Chemistry, Physics

• Direct Writing Assessment – Grades 5 and 8

11+ Point Structure for Elementary and Middle Schools

Overall School

600 Total Points

Growth 300 total

points

All Students200 total

points

Below Proficient Students 100 total

points

Achievement 300 total

points

Percent at or above

proficient 300 points

Schools without a 12th grade includes students in grades 3-8

12

Overall School

600 total points

Growth300 total

points

All students 200 total

points

Below Proficient Students 100 total

points

Achievement 300 total points

Percent at or above

proficient 150 total

points

Readiness Graduation

rate150 Total

points

Point Structure for High Schools

Schools with a 12th grade includes students in grades 9-12

+ 13Subgroups• Identifies below proficient students as a single

subgroup• Below Proficient Subgroup = All students who

scored below proficiency (level 1 or 2) on the previous year’s CRT

• Below Proficient Subgroup is determined independently for each content area (ELA, Math, Science)

• Ensures all students who are below standard, regardless of group, are the focus for improvement

• Below proficient subgroup is double weighted in the growth calculation to increase focus on those most at risk

• Complete disaggregated data for all 10 subgroups will be included in UCAS report including gap analysis

+ 14

Participation Requirement• A school must meet the 95% participation

rate for the whole school and non-proficient subgroups of 40 students or more in each content area

• Participation is calculated for the whole school and the non-proficient subgroup

• Schools not meeting the participation requirement will receive a UCAS total score of 0

15+ Sample ElementaryAchievement Calculation

Note: Schools without DWA , content areas are weighted equally (1/3 each)

Subject

Percent Proficien

t

PointsPossible(Weighted)

Achievement PointsEarned

ELA 80% x 8628.57% of

300

= 69

Math 70% x 8628.57% of

300

= 60

Science 60% x 8628.57% of

300

= 51

DWA 80% x 4314.29% of

300

= 34

Total 300 = 214

16+ High SchoolsCollege & Career Readiness College and Career Readiness accounts for 150 of

the 300 points for high schools in the achievement component.

The readiness component is the federal graduation rate calculation as approved by USED. All graduation reporting includes this rate.

For purposes of calculating UCAS, the graduation rate is calculated by multiplying the graduation rate by 150 (e.g. .70 x 150 = 105).

17+ Sample High School Achievement CalculationSubject

Percent Proficie

nt

PointsPossibl

e

Achievement Points Earned

ELA 80% x 5033% of

150

= 40

Math 70% x 5033% of

150

= 35

Science 60% x 5033% of

150

= 30

Readiness/Grad Rate (4 year federal)

80% x150

100% of 150

= 120

Total 300 = 225

+Additional Details

Student Growth

Percentile

18

SGP

19

Student Growth Percentile Student growth is determined by comparing the

performance of a student with all other students in the state with the same past scale score performance (1-3 years of CRTs). How does your gain compare to that of your like scoring peers? What is my percentile rank based on gain?

The Student Growth Percentile is also known as the “Colorado Growth Model”. It is used for accountability in the following states: Colorado, Nevada, Indiana, Massachusetts, New Hampshire, Maine, Arkansas, Rhode Island, Virginia, West Virginia, New York, Hawaii, Idaho, Georgia, Wyoming and Utah.

20+ NormativeHow does it work?Think of a group of students, where each student has two test scores – one for 2009 and one for 2010. We could show the distribution of these scores at the same time as pictured.

21+ NormativeWe could ‘slice’ through the picture to show the 2010 distribution for just one 2009 score. This is called a conditional distribution.The red shaded curve shows the conditional distribution in 2010 for all students who scored 166 in 2009.

22+ NormativeAssume we are interested in just one score, 170, in 2010. We could ask, what percentage of students who scored 166 in 2009 scored at or below a 170 in 2010?In this case, that turns out to be 75%. In other words, a score of 170 is at the 75th percentile.

SGP = 75

23

Why use SGP?Determines growth based on multiple years of

data for each studentHonors variable amounts of growth (including

small changes)Does not replicate proficiencyRecognizes growth for students who are

achieving at low and high ratesGrowth percentiles are calculated for every

student, but can be aggregated to the classroom, subgroup, school, district, and state

+

Additional DetailsCalculating

Growth24

25

New UCAS Growth Calculation

More finite point calculation vs. old table (2012) below.

ALL STUDENT Growth TABLEMedian

Growth Score Points Median Growth Score Points

70 or higher 200 50 12569 196 49 12168 193 48 11867 189 47 11466 185 46 11065 181 45 10664 178 44 10363 174 43 9962 170 42 9561 166 41 9160 163 40 8859 159 39 8458 155 38 8057 151 37 7656 148 36 7355 144 35 6954 140 34 6553 136 33 6152 133 32 5851 129 31 54

30 or Lower 50

Median SGP Achieved

All Students (Maximum 200

points)

0-34 50 35-49 100 50-59 150 60 and above 200

OLD Growth TABLE

26New UCAS Below Proficient Student Growth Calculation More finite point calculation vs. old table (2012) below.

BELOW PROFICIENT STUDENT Growth TABLE

Median SGP Achieved

Below Proficient Students

(Maximum 100 points)

0-34 25 35-49 50 50-59 75 60 and above 100

OLD Growth TABLE

Median Growth Score Points Median

Growth Score Points

70 or higher 100 50 6369 98 49 6168 96 48 5967 94 47 5766 93 46 5565 91 45 5364 89 44 5163 87 43 4962 85 42 4861 83 41 4660 81 40 4459 79 39 4258 78 38 4057 76 37 3856 74 36 3655 72 35 3454 70 34 3353 68 33 3152 66 32 2951 64 31 27

30 or Lower 25

    MGP PointsELA All Students Group 56 150 Below Proficient Group 35 50Math All Students Group 45 100  Below Proficient Group 55 75Science All Students Group 50 150  Below Proficient Group 40 50

Group

ELA Point

sMath Points

Science

PointsPoint Total

(mean)

All Students 150 100 150 133Below Proficient Students 50 75 50 58

Growth CalculationExample calculation

Total Growth Points

School Total Growth Points 133 + 58 = 191 ( 27 )

+

NEW CalculationsAnnual Measureable

Objectives 28

29Annual Measureable ObjectivesFederal Requirement to establish and report AMOsUtah’s Minimum Compliance Plan

AMOs are not used in any UCAS calculation AMO trajectory will reduce in half the percent of non-

proficient over six years AMOs will be established separately for each subgroup

at each school UCAS reporting will list the AMO and performance of

each school subgroup AMO reporting page will be a drill down page in the

UCAS report AMOs will be used in identifying and exiting Focus

schools

30+ Establishing AMOs AMOs will be based on the percent of students achieving

proficiency on the state’s Criterion-Referenced Tests (CRTs) separately in English language arts and mathematics.

ELA: CRT results in grades 3-8 and 10 are used to determine the percent of students proficient

Mathematics: results are based on CRTs in grades 3-6 and in the course appropriate CRT thereafter which includes math 7, algebra, or geometry for grades 7 and 8. High schools will be determined by calculating the percent of 10th grade students who scored proficient on the Algebra I CRT in 10th grade year or a prior year

Results from the Utah Alternative Assessment (UAA) are included for students with significant cognitive disabilities approved to participate in this assessment

31+AMO Sample Calculation for a School Subgroup with ELA Proficient = 82% 100% – 82% = 18% ½ of 18 is 9 9 / 6  years = 1.5 per year Year one   82.0 + 1.5 = 83.5 Year two   83.5 + 1.5 = 85.0 Year three 85.0 + 1.5 = 86.5 Year four  86.5 + 1.5 = 88.0 Year five  88.0 + 1.5 = 89.5 Year six   89.5 + 1.5 = 91.0  (half way to 100

percent)

AMO Sample Calculation

32+Sample AMO Trajectories for a School   All Asia

nAfricanAmer

.

Amer. Indian

 Hispanic

PacificIslande

r

 ED  LEP  SWD

2011

81

82 64 60 63 73 70 51 54

2012

83

84 67 63 66 75 73 55 58

2013

84

85 70 67 69 78 75 59 62

2014

86

87 73 70 72 80 78 63 66

2015

87

88 76 73 75 82 80 67 69

2016

89

90 79 77 78 84 83 71 73

2017

91

91 82 80 82 87 85 76 77

Goal:

91

91 82 80 82 87 85 76 77