measuring large-scale program impact on k-12 mathematics and science learning norman l. webb
DESCRIPTION
Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics and Science Partnership Evaluations Wisconsin Center for Education Research University of Wisconsin - PowerPoint PPT PresentationTRANSCRIPT
Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning
Norman L. WebbRobert H. MeyerPaula A. White
Adding Value to the Mathematics and Science
Partnership EvaluationsWisconsin Center for Education Research
University of Wisconsin
A Seminar at the University of California-Irvine
October 3, 2003
Historical Context:
Curriculum
1957-1972 New Math and Science Era—Curriculum Development
1972-1983 Back to the Basic and Withdrawal
Assessment
1972 California Assessment Program statewide committee (3,6,12)
1983 Nation At Risk
1986 38 states with state assessments
1978-1989 Accountability ERA—competency testing, state assessments
Standards
1989 NCTM Standards, Education Summit, National Goals
1991 Smith and O’Day, NSF State Systemic Initiatives
1994 45 states developing challenging standards
1996 26 SSIs, 16 USIs, 6 RSIs
2001 No Child Left Behind Act
2003 49 states with standards and assessments
National Longitudinal Study of Mathematical Abilities (NLSMA)
1962-1967
N=100,000 Students
Grade Levels
4 5 6 7 8 9 10 11 12
X Pop X X X X X
Y Pop X X X X X
Z Pop X X X
General Components of an Education System
Management
ProgramPolicy
Student Outcomes
System Attributes
Enabling
Target
Explanatory
AttributesEnabling
Capacity
Sustainability
Target
Alignment
Saturation
Quality
Equity
Linkages
Explanatory
Trade-offs
Incentives
Coordinated Linked Capacities Lead to System Fitness for Change
System Component Elements of Components
1. Pre-SSI Policy •Standards/Frameworks•Professional Development•Assessment•Accountability•Other Instructional Guidance Policies
2. Pre-SSI Infrastructure Networks of•Policymakers•Professional Developers•School Administrators•School Restructuring Organizations•K-12 Teachers of Mathematics and Science
3. SSI Leadership and Change Strategy
•Educational Change Strategy•Organizational Change Strategy
4. SSI Policy •Standards/Frameworks•Professional Development•Assessment •Accountability•Other Instructional Guidance Policies
Components and Elements of Systemic Reform
System Component Elements of Components
5. SSI Infrastructure Networks of•Policymakers•Professional Developers•School Administrators•School Restructuring Organizations•K-12 Teachers of Mathematics and Science
6. SSI Standards-Based Instructional Reform
•Individual Capacity Building•Organizational Capacity Building•Classroom Practice
7. Student Outcomes •Significant Statewide Gain•Substantial Statewide Gain•Standards-Based Gain•Control Group Gain•Gap Closing
Components and Elements of Systemic Reform (Continued)
Operationalizing Elements of Systemic Reform (Continued)
Example 1: SSI Policy (and Pre-SSI Policy) Component
Element Breadth (1 to 5) Depth (Low = 1: High = 5)
Prior to the SSI, state policies were in place for mathematics/science professional development for 1 = one relevant group of actors (e.g., school administrators; elem., middle, or highschool in-service teachers of math/science; pre-service teachers of math/science)3 = three of above groups5 = all above groups
State policy on mathematics/science professional development is 1 = minimally 3 = substantially 5 = highly• aligned to state curriculum standards/frameworks• prescriptive (i.e., specific about what teachers need to know and be able to do to successfully engage all students in standards-based learning, and specific about professional development practices that engage teachers in necessary learning)• authorized (e.g., by professional organizations for K-16 mathematics/science, SEA)• supported by powerful incentives and sanctions (i.e., participating in or ignoring professional development has significant direct, concrete consequences for teachers, administrators, schools, or districts)
Pro
fess
iona
l De v
e lo p
men
t
0
1
2
3
4
5
0 1 2 3 4 5 6Breadth
De
pth
Standards/Framework
Professional Development
Assessment
0
1
2
3
4
5
0 1 2 3 4 5 6Breadth
De
pth
Standards/FrameworkProfessional DevelopmentAssessmentClassroom Practice
Pre - SSI SSI
Figure 10. Breadth and depth assessment for selected SSI components: Arkansas.
Pre - SSI SSI
0
1
2
3
4
5
0 1 2 3 4 5 6Breadth
De
pth
Standards/Framework
Professional Development
Assessment
0
1
2
3
4
5
0 1 2 3 4 5 6
BreadthD
epth
Standards/FrameworkProfessional DevelopmentAssessmentClassroom Practice
Figure 11. Breadth and depth assessment for selected SSI components: Connecticut.
0
1
2
3
4
5
0 1 2 3 4 5 6Breadth
De
pth
Standards/Framework
Professional Development
Assessment
0
1
2
3
4
5
0 1 2 3 4 5 6Breadth
De
pth
Standards/FrameworkProfessional DevelopmentAssessmentClassroom Practice
Pre - SSI SSI
Figure 12. Breadth and depth assessment for selected SSI components: Louisiana.
Focus of State SSI and Statewide Achievement Gains from 1992 to 2000
State System
and InfrastructureBalanced
Close to the Classroom
Steady IncreaseMichigan
TexasLouisiana
MassachusettsKentucky
New York
Some Increase
Connecticut
Georgia
South Carolina
Arkansas
Little/No Change MaineCalifornia Nebraska
New Mexico
Basic Evaluation Model for Judging the Impact of a MSP
ΔAchievement = f (MSP) + ε
SCALE Evaluation Indicator System
Context
Project SCALE
District School
Input
Teacher
StudentDispositionTo Learn
Opportunity To Learn
StudentAchievement
Capacity Action Student Outcomes
StudentParticipation
University
Number of Hispanic Students Tested by Cohort and Year
Number of White Students Tested by Cohort and Year
Rasch Scale, EV1 Model
-2.000
-1.500
-1.000
-0.500
0.000
0.500
1.000
1.500
2.000
2.500
1995 1996 1997 1998 1999 2000
Year
US
I E
ffe
ct
grade 4 grade 5 grade 6 grade 7 grade 8
Effectiveness of the Urban Systemic Initiative by Grade and Year for the Rasch Scale and EV1 Model
From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between
White and Black Students
-3.00
-2.50
-2.00
-1.50
-1.00
-0.50
0.00
0.50
1995 1996 1997 1998 1999 2000
Year
Eff
ec
t o
f A
fric
an
Am
eri
can
4th
5th
6th7th8th
Average
Rasch Scale (EV1 Estimates)
From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between
White and Hispanic Students
-2.50
-2.00
-1.50
-1.00
-0.50
0.00
0.50
1.00
1995 1996 1997 1998 1999 2000
Year
Eff
ect
of
His
pa
nic
4th
5th6th
7th
8th
Average
Rasch Scale (EV1 Estimates)
-3.00
-2.50
-2.00
-1.50
-1.00
-0.50
0.00
0.50
1995 1996 1997 1998 1999 2000
Year
Eff
ect
of E
co
nom
ic D
isad
van
tag
e
8th
7th
4th
6th
5th
Average
From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between
Advantaged and Disadvantaged Students
(EV1 Estimates)
Evaluation Design Indicator System Experimental Design Quasi-experimental Design Qualitative Analysis Focus Study
Experimental Design Control Group Random Assignment Post Measure Sufficient N for Power Requirements
Quasi-Experimental Design
Comparison Group Pre-Post Measures Replications
Independent Variables Teacher Knowledge Teacher Practice Curriculum Student Activities Support Resources Professional Development
SCALE Evaluation Indicator System
Context
Project SCALE
District School
Input
Teacher
StudentDispositionTo Learn
Opportunity To Learn
StudentAchievement
Capacity Action Student Outcomes
StudentParticipation
University