1

School Evaluation ServicesPennsylvania Update

Education Policy and Leadership CenterWestern Pennsylvania Breakfast Series

Jonathan Jacobson, Director

May 23, 2002

2

This Morning’s Presentation

SES Background

Status Update

User Feedback

Enhancements

Pennsylvania Statewide Insights

What’s Next

3

SES Background

4

What is School Evaluation Services?

A powerful diagnostic and decision-making tool for administrators, educators, policymakers, taxpayers, and parents

Annual written report on the strengths, challenges, and risks for each of the state’s school districts

Public website containing these reports and detailed supporting data trends and comparative benchmarks

Annual comprehensive report on the state’s collective circumstances, yearly progress, and policy implications

5

SES converts data to information through its unique analytical framework

Transparency, consistency, accessibility

Objective interpretation

“Return on Resources” – simultaneous examination of spending, results, and context

Multi-year trends measure progress

Each district compared to five benchmarks: state, I.U., county, and demographic peer averages, as well as district itself over time

Website’s interactive display of data allows users to customize comparison groups

6

Five comparison groups to choose from

SES website data menu example – Student Results

Six data subsections to choose from

7

How does SES help Pennsylvania’s education stakeholders?

SES goes beyond reporting test scores – it synthesizes achievement indicators with financial data to measure return and inform resource allocation decisions

There may not be sufficient resources or staff at either the state or the local level to perform data benchmarking and analysis – SES fills this void

SES helps decision-makers at all levels avoid being “data rich but information poor”

SES helps end policymakers’ and stakeholders’ reliance on anecdotal generalizations

8

Pennsylvania Advisory Group

Leslye Abrutyn, Superintendent, Penn-Delco SD

Carolyn Dumaresq, Executive Director, PSEA

Tom Gentzel, Executive Director, PSBA

Jay Himes, Executive Director, PASBO

Sheri Rowe, Office of Educational Initiatives, PDE

Stinson Stroup, Executive Director, PASA

Larry Wittig, State Board of Education, State Workforce Investment Board

Jan Zastawniak, President, PenSPRA

9

SES Status Update

10

Milestones accomplished

SES launched for Michigan – May 2001

SES launched for Pennsylvania – October 2001

Interim data update for Michigan – October 2001

Interim data update for Pennsylvania – November 2001

“Statewide Insights” released for Michigan – December 2001

Round 2 released for Michigan – January 2002

“Statewide Insights” released for Pennsylvania – May 2002

11

Milestones to come

Round 2 (2000 data) release for Pennsylvania – May 2002

Round 3 (2001 data) release for Pennsylvania – 4Q 2002

12

SES website traffic has exceeded expectations

May

2001

Oct

2001

Jan

2002

March

2002

Cumulative

(through 05/04/2002)

Visitor Sessions

24,724 88,740 50,695 71,514 522,307

Page Views 1,620,819 3,890,785 808,288 875,897 13,510,955

13

Parents are the most frequent visitors to the SES website

Parents 39%

Teachers 18%

School Administrators 11%

Taxpayers 10%

School Board Members 3%

Government Officials 2%

Other 17%

Pennsylvania data through 04/30/2002

14

Usage among school personnel is significant

Michigan Pennsylvania

School Districts 554 501

Administrator Log-Ins 18,832 10,746

School Teacher Log-Ins 18,035 18,258

Cumulative data through 04/30/2002

15

While activity peaks predictably during launch periods, visits during non-launch periods are relatively constant

Average Daily “Off-Peak” Logins: 233

PA population is 12 million; MI population is 9.8 million

Daily Site Activity for MI

0

2000

4000

6000

6/2

1

7/2

1

8/2

1

9/2

1

10

/21

11

/21

12

/21

1/2

1

2/2

1

3/2

1

Lo

gin

sDaily Site Activity for PA

0

1000

2000

3000

4000

11

/6

11

/20

12

/4

12

/18

1/1

1/1

5

1/2

9

2/1

2

2/2

6

3/1

2

3/2

6

4/9

Lo

gin

s

Average Daily “Off-Peak” Logins: 331

16

Pennsylvania activity has nearly matched Michigan’s in much less time

Michigan Pennsylvania

Launch Date May 25, 2001

(11 months)

October 3, 2001

(7 months)

Share of Total Log-Ins 52% 48%

Cumulative data through 04/30/2002

17

SES User Feedback

18

Standard & Poor’s seeks, values, and uses customer feedback

E-mail links, state-designated e-mail boxes, and a toll-free telephone number enable easy and frequent communication

All inquiries, comments, and suggestions are logged, identified by type, distributed, and tracked

Responses are timely, thorough, and courteous

More formal research is underway

User feedback is a primary source of SES enhancements

19

Positive commentary about SES repeatedly cited by Pennsylvania school leaders Helpful in strategic planning, budgeting, benchmarking

Comparative data are useful—particularly peers

Effective accountability tool

Useful in communicating to school board, local stakeholders

Information has prompted discussions in community

Helpful for people considering moving into community

Coming from an objective source makes SES more valuable

20

Other positive comments about SES from school administrators

Helped achieve programmatic improvements

Used SES data to renegotiate benefits package—cost savings were converted to teacher salary increases

Potential to use SES data to identify best practices

“Return on resources” concept is innovative and valuable

Explanations, disclaimers, caveats are helpful

Service is needed—grateful that it’s being provided

21

SES Enhancements

22

Many improvements have been implemented since the initial launch of the SES website last year Major enhancements

Intermediate Unit comparison group

Interim statewide data revisions

Display of basic school-level data

“Beating the odds” schools in S&P Observations

Statewide graphics

Navigation menu page

Enhanced district trend data display

23

24

25

Many improvements have been implemented since the initial launch of the SES website last year Other enhancements

Expansion of S&P Observations to include methodology, glossary, and list of peers

“State homepage” – administrator’s toolkit, state education department links, state facts

Navigation path icons – clearer indication of actions and selections

“Contact Us” e-mail links

Homepage content and reconfiguration

Log In page simplification

26

Additional enhancements are in the works

Data display

School-level information

Data sorting tools

Format and organization of S&P Observations

User interface

School/district searching tools

Website tutorial/presentation

Data mapping explanation/crosswalk

27

28

Pennsylvania Statewide Insights

29

PSSA indicators show only slight improvement

Highest district PSSA Composite Score was 1457, while the lowest was 1087

Highest district percent of scores above median was 80.5%, while the lowest was 6.3%

Student Results

30

Across the state, white students’ PSSA scores were significantly higher than those of black, Hispanic, Native American, and multi-ethnic students

Achievement Gaps

31

Return on Resources

PSSA results have increased slightly, but so has spending (net of inflation) – therefore, the state’s return on resources has not improved

Performance Cost Index (PCI) data suggest that reasonable spending increases are not likely to make enough difference by themselves, if the goal is to enable all students to meet state standards

A better rate of return on resources is needed if additional spending is to make a substantial impact

Strategies to improve achievement may need to include, but go well beyond, monetary inputs

A comparison of data for specific school districts leads to the same conclusion…

32

There is no significant correlation between spending and results – at any given spending level, there is a wide range of student achievement Pennsylvania School Districts

1999 Per-Pupil Spending and PSSA Scores

1000

1050

1100

1150

1200

1250

1300

1350

1400

1450

1500

4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000

Operating Expenditures Per Student ($)

PS

SA

Me

an

Co

mp

osi

te S

co

re

Philadelphia: $7,194 PSSA Score = 1134

State Weighted Average Score = 1303

State Weighted Average Spending = $7,367 Per Pupil

Pittsburgh: $9,846 PSSA Score = 1224

Harrisburg: $9,040 PSSA Score = 1106

Return on Resources (continued)

33

Spending “decreases” for Philadelphia, Harrisburg, and Pittsburgh when adjusted for the higher cost of serving students with special needs, and for geographic differences in the cost of education

Pennsylvania School Districts 1999 Adjusted Spending (NCOL & Student Circumstances) and PSSA Scores

1000

1050

1100

1150

1200

1250

1300

1350

1400

1450

1500

4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000 Operating Expenditures Per Student Adjusted for NCOL and Student Circumstances ($)

PS

SA

Mea

n C

om

po

site

Sco

re

Philadelphia: $4,902 PSSA Score = 1134

State Weighted Average Expenditure = $6,828 Per Pupil

State Weighted Average Score = 1303

Pittsburgh: $8,229 PSSA Score = 1224

Harrisburg: $7,323 PSSA Score = 1106

Return on Resources (continued)

34

A number of districts with above-average enrollments of economically disadvantaged students demonstrated a better-than-average return on resources, as measured by lower (more favorable) PCIs.

Pennsylvania School Districts

Performance Cost Index (PSSA) and Low-Income Enrollment

0

10

20

30

40

50

60

70

80

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Enrollment of Low-Income Students (%)

Pe

rfo

rma

nc

e C

os

t In

de

x f

or

PS

SA

Me

an

Co

mp

os

ite

S

co

res

($

)

Weighted Avg. = $22.51

Weighted Avg. = 31.7%

24 Districts(14 districts had PSSA results that were at or above the average)

153 Districts

151 Districts

172 Districts 4 Districts are not shown in this quadrant because they are outliers

Philadelphia

Pittsburgh

Harrisburg

Return on Resources (continued)

35

PSSA scores and socioeconomic status are correlated; however, many schools “beat the odds”

Pennsylvania Elementary SchoolsPSSA Scores and Low-Income Enrollment

800

850

900

950

1000

1050

1100

1150

1200

1250

1300

1350

1400

1450

1500

1550

1600

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Enrollment of Low-Income Students (%)

PS

SA

Me

an

Co

mp

osi

te S

core

Weighted Avg. = 1314

Weighted Avg. = 35.4%

163 Elementary Schools Beat the Odds

481 Schools

707 Schools

153 Schools

Best Practices

36

Many school districts as a whole also “beat the odds”

Pennsylvania School DistrictsPSSA Scores and Low-Income Enrollment

900

950

1000

1050

1100

1150

1200

1250

1300

1350

1400

1450

1500

1550

1600

0 10 20 30 40 50 60 70 80 90 100

Enrollment of Low-Income Students (%)

PS

SA

Me

an

Co

mp

os

ite

Sc

ore

Weighted Avg. = 1303

Weighted Avg. = 31.7%

248 Districts

75 Districts

46 Districts Beat the Odds

10 districts remained in this quadrant for 3 consecutive years

131 Districts

Best Practices (continued)

37

What do the data tell us?

No “silver bullets”

Without improved return, goals of improving student achievement overall and of closing achievement gaps are probably unaffordable

How money is spent is as important as how much money is spent

Need to align policy making with data analysis

Districts and schools that beat the odds – particularly those that do so consistently over time – are potential sources of best practice

38

Next Steps

39

Coming with next release…

Charter school information

School-level data

Peer methodology to include state aid ratio

More detailed trend data

Improved location search features

Website self-tutorial

Data mapping crosswalk

40

Ongoing efforts…

Tutorial and training programs

More detailed school-level information

Data sorting tools

Market research and user outreach

41

Where to find us, how to reach us

Website URL: www.ses.standardandpoors.com

E-mail address: ses_pa@standardandpoors.com