measuring large-scale program impact on k-12 mathematics and science learning  norman l. webb

Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning

 Norman L. WebbRobert H. MeyerPaula A. White

 Adding Value to the Mathematics and Science

Partnership EvaluationsWisconsin Center for Education Research

University of Wisconsin

  A Seminar at the University of California-Irvine

October 3, 2003

Historical Context:

Curriculum

1957-1972 New Math and Science Era—Curriculum Development

1972-1983 Back to the Basic and Withdrawal

Assessment

1972 California Assessment Program statewide committee (3,6,12)

1983 Nation At Risk

1986 38 states with state assessments

1978-1989 Accountability ERA—competency testing, state assessments

Standards

1989 NCTM Standards, Education Summit, National Goals

1991 Smith and O’Day, NSF State Systemic Initiatives

1994 45 states developing challenging standards

1996 26 SSIs, 16 USIs, 6 RSIs

2001 No Child Left Behind Act

2003 49 states with standards and assessments

National Longitudinal Study of Mathematical Abilities (NLSMA) 

1962-1967 

N=100,000 Students 

Grade Levels

4 5 6 7 8 9 10 11 12

X Pop X X X X X

Y Pop X X X X X

Z Pop X X X

General Components of an Education System

Management

ProgramPolicy

Student Outcomes

System Attributes

Enabling

Target

Explanatory

AttributesEnabling

Capacity

Sustainability

Target

Alignment

Saturation

Quality

Equity

Linkages

Explanatory

Trade-offs

Incentives

Coordinated Linked Capacities Lead to System Fitness for Change

System Component Elements of Components

1. Pre-SSI Policy •Standards/Frameworks•Professional Development•Assessment•Accountability•Other Instructional Guidance Policies

2. Pre-SSI Infrastructure Networks of•Policymakers•Professional Developers•School Administrators•School Restructuring Organizations•K-12 Teachers of Mathematics and Science

3. SSI Leadership and Change Strategy

•Educational Change Strategy•Organizational Change Strategy

4. SSI Policy •Standards/Frameworks•Professional Development•Assessment •Accountability•Other Instructional Guidance Policies

Components and Elements of Systemic Reform

System Component Elements of Components

5. SSI Infrastructure Networks of•Policymakers•Professional Developers•School Administrators•School Restructuring Organizations•K-12 Teachers of Mathematics and Science

6. SSI Standards-Based Instructional Reform

•Individual Capacity Building•Organizational Capacity Building•Classroom Practice

7. Student Outcomes •Significant Statewide Gain•Substantial Statewide Gain•Standards-Based Gain•Control Group Gain•Gap Closing

Components and Elements of Systemic Reform (Continued)

Operationalizing Elements of Systemic Reform (Continued)

Example 1: SSI Policy (and Pre-SSI Policy) Component

Element Breadth (1 to 5) Depth (Low = 1: High = 5)

Prior to the SSI, state policies were in place for mathematics/science professional development for 1 = one relevant group of actors (e.g., school administrators; elem., middle, or highschool in-service teachers of math/science; pre-service teachers of math/science)3 = three of above groups5 = all above groups

State policy on mathematics/science professional development is 1 = minimally 3 = substantially 5 = highly• aligned to state curriculum standards/frameworks• prescriptive (i.e., specific about what teachers need to know and be able to do to successfully engage all students in standards-based learning, and specific about professional development practices that engage teachers in necessary learning)• authorized (e.g., by professional organizations for K-16 mathematics/science, SEA)• supported by powerful incentives and sanctions (i.e., participating in or ignoring professional development has significant direct, concrete consequences for teachers, administrators, schools, or districts)

Pro

fess

iona

l De v

e lo p

men

t

0

1

2

3

4

5

0 1 2 3 4 5 6Breadth

De

pth

Standards/Framework

Professional Development

Assessment

0

1

2

3

4

5

0 1 2 3 4 5 6Breadth

De

pth

Standards/FrameworkProfessional DevelopmentAssessmentClassroom Practice

Pre - SSI SSI

Figure 10. Breadth and depth assessment for selected SSI components: Arkansas.

Pre - SSI SSI

0

1

2

3

4

5

0 1 2 3 4 5 6Breadth

De

pth

Standards/Framework

Professional Development

Assessment

0

1

2

3

4

5

0 1 2 3 4 5 6

BreadthD

epth

Standards/FrameworkProfessional DevelopmentAssessmentClassroom Practice

Figure 11. Breadth and depth assessment for selected SSI components: Connecticut.

0

1

2

3

4

5

0 1 2 3 4 5 6Breadth

De

pth

Standards/Framework

Professional Development

Assessment

0

1

2

3

4

5

0 1 2 3 4 5 6Breadth

De

pth

Standards/FrameworkProfessional DevelopmentAssessmentClassroom Practice

Pre - SSI SSI

Figure 12. Breadth and depth assessment for selected SSI components: Louisiana.

Focus of State SSI and Statewide Achievement Gains from 1992 to 2000

State System

and InfrastructureBalanced

Close to the Classroom

Steady IncreaseMichigan

TexasLouisiana

MassachusettsKentucky

New York

Some Increase

Connecticut

Georgia

South Carolina

Arkansas

Little/No Change MaineCalifornia Nebraska

New Mexico

Basic Evaluation Model for Judging the Impact of a MSP

 ΔAchievement = f (MSP) + ε

SCALE Evaluation Indicator System

Context

Project SCALE

District School

Input

Teacher

StudentDispositionTo Learn

Opportunity To Learn

StudentAchievement

Capacity Action Student Outcomes

StudentParticipation

University

Number of Hispanic Students Tested by Cohort and Year

Number of White Students Tested by Cohort and Year

Rasch Scale, EV1 Model

-2.000

-1.500

-1.000

-0.500

0.000

0.500

1.000

1.500

2.000

2.500

1995 1996 1997 1998 1999 2000

Year

US

I E

ffe

ct

grade 4 grade 5 grade 6 grade 7 grade 8

Effectiveness of the Urban Systemic Initiative by Grade and Year for the Rasch Scale and EV1 Model

From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between

White and Black Students

-3.00

-2.50

-2.00

-1.50

-1.00

-0.50

0.00

0.50

1995 1996 1997 1998 1999 2000

Year

Eff

ec

t o

f A

fric

an

Am

eri

can

4th

5th

6th7th8th

Average

Rasch Scale (EV1 Estimates)

From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between

White and Hispanic Students

-2.50

-2.00

-1.50

-1.00

-0.50

0.00

0.50

1.00

1995 1996 1997 1998 1999 2000

Year

Eff

ect

of

His

pa

nic

4th

5th6th

7th

8th

Average

Rasch Scale (EV1 Estimates)

-3.00

-2.50

-2.00

-1.50

-1.00

-0.50

0.00

0.50

1995 1996 1997 1998 1999 2000

Year

Eff

ect

of E

co

nom

ic D

isad

van

tag

e

8th

7th

4th

6th

5th

Average

From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between

Advantaged and Disadvantaged Students

(EV1 Estimates)

Evaluation Design Indicator System Experimental Design Quasi-experimental Design Qualitative Analysis Focus Study 

Experimental Design Control Group Random Assignment Post Measure Sufficient N for Power Requirements 

Quasi-Experimental Design

Comparison Group Pre-Post Measures Replications

Independent Variables Teacher Knowledge Teacher Practice Curriculum Student Activities Support Resources Professional Development

SCALE Evaluation Indicator System

Context

Project SCALE

District School

Input

Teacher

StudentDispositionTo Learn

Opportunity To Learn

StudentAchievement

Capacity Action Student Outcomes

StudentParticipation

University