1

California's API:Recent Developmentsand Future Prospects

Edward H. Haertel

Stanford UniversitySchool of Education

CRESST Conference University of California at Los Angeles

September 10, 2004

2

The "API" is California's

Academic Performance Index• School-level summary of student test

performance

• Mandated by Public Schools Accountability Act of 1999 (PSAA)

• Foundation of California's (pre-NCLB) public-school accountability system

3

Plan for This Morning

• Calculating the API

• Coping with change

• Lessons learned

4

Calculating the API

5

Calculating the API in 1999It used to be so simple...

• Index based just on SAT-9 scores

• Elem/Middle: Reading, Language Arts, Spelling, Math

• High School: Reading, Language Arts, Math, Science, Social Studies

6

Subject Area Weights

•Reading 30%

•Lang Arts 15%

•Spelling 15%

•Math 40%

•Reading 20%

•Lang Arts 20%

•Math 20%

•Science 20%

•Soc St 20%

Elem/Middle High School

7

Precursors to Standards-Based Performance Levels

•1-19

•20-39

•40-59

•60-79

•80-99

•200

•500

•700

•875

•1000

SAT-9 National Percentile Rank

"Progressive Weighting Factor"

8

Performance Levels on California Standards Tests (CSTs)

•Far Below Basic

•Below Basic

•Basic

•Proficient

•Advanced

•200

•500

•700

•875

•1000

CST Performance Level "Weighting Factor"

9

Growth Targets

• Statewide Perf. Target = 800

• Annual Growth Target =5% of distance from Base-Year API to 800

• Initially, schools at/above 800 need only remain at that level

10

Numerically Significant Subgroups

• "Comparable Improvement" require-ment mandated by PSAA of 1999

• Groups defined by Race/Ethnicity and by socioeconomic disadvantage

• Student in soc. dis. group if either (1) eligible for free/reduced price meal program or

(2) highest parent educational level is "not a high school graduate."

11

Defining "Numerically Significant"

• How large must a subgroup be to count as numerically significant?

• Fewer than 30 students not num. sig.

• 30-99 students num. sig. only if subgroup constitutes at least 15% of total enrollment

• 100 or more students are num. sig.

• For purposes of API growth calculation, subgroup must be num. sig. in both base year and growth year

12

Defining "Comparable Improvement"

• Growth target is defined in terms of difference between subgroup's score in base year and in growth year

• Subgroup growth target is 80% of schoolwide target

13

Coping with Change

14

"IF YOU WANT TO

MEASURE CHANGE,

DON'T CHANGE THE

MEASURE."--Al Beaton

15

Base API and Growth API

May ... Oct ... Feb ... May ... Oct ... Feb ...

testing Growth Base testing Growth Base API API API API Release Release Release Release

School Year School Yeartesting cycle testing cycle

Biannual Accountability Cycle

16

The "Scale Calibration Factor"

• The SCF is a constant added to both the Base API and the Growth API within an accountability cycle.

• It is chosen to make the mean Base API for that cycle equal to the mean Growth API for the previous cycle.

• There are new SCFs each year for Elementary schools, Middle schools, and High schools

17

The "Scale Calibration Factor"

May ... Oct ... Feb ... May ... Oct ... Feb ...

testing Growth Base testing Growth Base API API API API Release Release Release Release

School Year School Yeartesting cycle testing cycle

Adjusted to equal state-level means

via SCF

Adjusted to equal state-level means

via SCF

18

Changes in the Measure

• Changes in the tests included

• Changes to accommodate heterogeneity of California schools

• Evolving decision rules for special situations

• Changes due to NCLB Act of 2001

• Refinements in data acquisition/ quality control

19

Changes in the Tests Included

• 1999-00 SAT-9 only

• 2000-01 SAT-9 only

• 2001-02 add California StandardsTest (CST) in ELA

• 2002-03 add CST-Math (G2-11);add CST-Hist/SocSt (G10-11); drop SAT-9 Soc St; add CAHSEE G9-10

20

Changes in the Tests Included• 2003-04 add CST-Science (G9-11);

incorporate CAPA;CAHSEE now G10-11;add CAT-6; drop SAT-9

• 2004-05 ??? (2004-05 Base APIto be posted March

2005)

21

Changes in the Tests Included

Future changes will include:

• reducing CAT-6 to brief survey at just two or three grade levels

• adding CST-Science (G5)

• adding cumulative Hist/Soc Sci (G8)

• expansion of CAHSEE to include G11, G12 retests

• ... ?

22

Heterogeneity of California Schools

• Schools that cut across usual Elem/Middle/High boundaries

• Small and very small schools

• Alternative School Accountability Model (ASAM) schools

23

Most Common Grade Spans for California Schools

K-6 1857 10-12 160 K-5 1800 6-8 & Ungraded 9-12 1067 Elementary 157 6-8 653 K-6 & Ungraded K-5 & Ungraded Elementary 154 Elementary 586 7-12 139 K-8 547 K-12 108 7-8 332 K-3 101

Data from "Accountability Update – Spring 2004" at http://www.cde.ca.gov/ta/ac/ay/

24

Small (and Very Small)and ASAM Schools

• First, schools with N < 100 did not receive API

• Then, schools with N between 11 and 99 received API with an asterisk

• Now, due to inclusion of API as alternative indicator in California's NCLB accountability, all schools must receive API, including ASAM schools and those with N < 11

25

Evolving Decision Rules

• denominator (total enrollment) for testing participation rate calculation

• Parent opt-outs

• students tested with

• Testing accommodations

• Testing modifications

• California Alternate Performance Assessment (CAPA)

26

Evolving Decision Rules

• Testing irregularities

• Student testing irregularities

• Adult testing irregularities

• Decision rules for deciding which has occurred

• Appeals processes

• Growth calculations when API is invalidated for one year

27

Evolving Decision Rules

• Mobility Exclusions

• Students new to district

• Students new to school

• Special rules for "feeder school" patterns

• Non-universal indicators

• High School Students not enrolled in a math or science course

• Changing rules for out-of-level testing

28

Evolving Decision Rules

• Partial records

• Counting students taking some but not all subtests

• G4 or G7 writing tests with no CST-ELA

• blank test forms

• Students taking high school math or science courses earlier/later than standard grade sequence

29

Impact of NCLB...

... has been massive, requiring changes in what subgroups are included, how participation rate is calculated (exclusions, averaging), what schools get APIs, what tests are given at each grade, rules for out-of-level testing, data release timelines, design of score reports, treatment of students designated proficient on CAPA, ...

30

Data Acquisition/Quality Control

• School-level analog to standardized testing conditions at student level

• Requires looking at API as product of complex system, dependent on

• Government officials, State Board of Ed

• Actors in school and district offices

• Testing contractors

• Teachers

31

Data Acquisition/Quality Control

• Data sources have changed over time for

• Total school enrollment numbers

• Free/Reduced Price Meal Participation numbers

• New "2004 STAR Pre-Identification Data Review"

32

Lessons Learned

33

Lessons Learned

• Long-term stability is probably unattainable

• Complexity increases, and increases, and increases

• Technical and policy considerations often point to divergent courses of action

34

Lessons Learned

• Seeing Like a State*

• Fairness is a matter of degree

• Comparison of API (successive cohort) vs. matched individual growth indices

• Inequities in accounting for educational challenge

• Paradoxical effects

*Title of 1998 book by James C. Scott

35

Lessons Learned• With hard work, a system can be

created that is trusted and respected, despite its imperfections

• By and large, scores improved steadily over the entire time California used the SAT-9*

• An enormous "behind the scenes" effort is required to maintain a reliable and valid accountability system

*See David Rogosa's paper, "Four-peat: Data Analysis Results from Uncharacteristic Continuity in California Student Testing Programs" at http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

36

What Can Policy Makers Do?• Create and use a stable technical

advisory committee

• Plan ahead, but stay flexible

• Resist pressures to use tests for new and different purposes

• Remember that the school-level index is only part of an accountability system and only part of an education indicator system