teacher value-added reports state board of education january 14, 2014 jamie meade managing director,...

TEACHER VALUE-ADDED REPORTSSTATE BOARD OF EDUCATIONJANUARY 14, 2014

Jamie MeadeManaging Director, Strategic MeasuresBattelle for Kids

© 2013, Battelle for Kids. All Rights Reserved.

Session Objectives

Provide value-added information relevant to the practitioner’s perspective. What should teachers and administrators know

and understand about value-added reporting?

How may educators use value-added information to improve professional practice and impact student academic achievement and progress?


Value-Added in Ohio

For over 10 years, Battelle for Kids has provided support for professional learning and fostered collegial dialogue for understanding and using value-added measures.

Battelle for Kids continues to advocate for the use of value-added measures, in combination with other educational measures, to improve practice and accelerate student academic progress.


Ohio’s Value-Added History

2002: Battelle for Kids’ SOAR Collaborative School and District Value-Added Reports

2006: Battelle for Kids’ TCAP Project Teacher Value-Added Reports

2007: ODE Value-Added on Local Report Card 2009: Battelle for Kids’ Ohio VA High Schools 2011: RttT: 30% Teachers, VA Reports 2012: RttT: 60% Teachers, VA Reports 2013: 4-8 Reading & Math Teachers, VA Reports

80 RttT Districts, K-3 R & M Teachers, VA Reports


Value-Added in Ohio

Educational Value Added Assessment System (EVAAS)

SAS Analytics Customers in 135 countries More than 65,000 business, government

and university sites SAS customers or their affiliates represent

90 of the top 100 companies on the 2012FORTUNE Global 500® list.


Value-Added Information in Practice:Building Awareness

Understanding the Difference Achievement Measures Progress / Growth Measures


Achievement & Growth:Understanding the Difference

Achievement

Growth


Measuring Growth is Important for ALL Students

Proficient

Grade

3rd 4th 5th 6th 7th 8th

Jacob

Adam


“Why can’t we simply compare OAA scaled scores from one year to the next to measure growth?”

Level 3rd 4th 5th 6th 7th 8th

Advanced 447-521452-551 439-548 448-575 458-574 459-537

Accelerated 429-446432-451 424-438 429-447 436-457 432-458

Proficient 400-428400-431 400-423 400-428 400-435 400-431

Basic 378-399377-399 382-399 378-399 378-399 379-399

Limited 240-377239-376 245-381 235-377 268-377 288-378

OAA Math Scaled Score Ranges May 2013


Questions?


Value-Added Information in Practice: SAS EVAAS MRM Model

Mutlivariate Response Model (MRM) Grades 4–8 Reading and Math VA Reports Uses Ohio Achievement Assessments


SAS EVAAS Value-Added MRM Model*

Uses Grades 3–8 Reading and Math OAAs. Compares the average growth of students in the

most recent year to the average growth of students in 2010 (state’s baseline year)*

Growth expectation is defined as maintaining placement in the distribution of NCE scores from one year to the next*

*conceptual definition


Raw Score Scaled Score NCESample

Raw Score Range

Sample Scaled Score

Range

Normal Curve Equivalent

NCE

52

0

551

247

99

1


Scaled Scores Converted to NCEs in State’s 2010 Baseline Year

Example2010 Scaled Scores

Rank Ordered551

247

Normal Curve Equivalent (NCE)

99

1

Conversion Values Are Fixed / Frozen


Value-Added Terminology


The NCE is similar to a percentile rank in that scores are derived from scaled scores and ranked based upon performance.

A significant difference between percentile rank and NCE is that an NCE scale is an equal interval scale.



1 10 20 30 40 50 60 70 80 90 99

1 5 10 20 30 40 50 60 70 80 90 95 99

Distributionof Scores

Normal CurveEquivalents

PercentileEquivalents



Baseline Score Group of students’ prior year mean NCE Example: Spring 2012 OAA mean NCE

Observed Score Group of students’ new/most recent mean NCE Example: Spring 2013 OAA mean NCE


SAS EVAAS MRM ModelBasic, Conceptual Example

Baseline OAA 2012

99th NCE

Student 255 NCE

Student 392 NCE

Student 120 NCE

Observed OAA 2013

99th NCE

Student 267 NCE (+12)

Student 120 NCE (0)

Expected Growth

Maintain Placement in Distribution of

Scores

For example, a student at the 20th

NCE must “at least” stay at the 20th NCE

Student 389 NCE (-3)


Basic, Conceptual Example

Scaled scores are converted to NCEs

394 = 46 Student 1 430 = 59

402 = 50 Student 2 417 = 54

384 = 42 Student 3 400 = 49

394 = 46 Student 4 390 = 44

410 = 52 Student 5 425 = 57

Mean Baseline = 47.2

Grade 6Baseline

Grade 7Observed

Mean Observed = 52.6

A basic measure of the growth for this group is 5.4 NCEs

Growth = Mean Observed – Mean Baseline

Growth = 52.6 – 47.2 = 5.4 (Mean NCE Gain)


Questions?


Levels of Value-Added Effects

Students are making substantially more progress than the state growth standard.

Students are making more progress than the state growth standard.

Students are making about the same amount of progress as the state growth standard.

Students are making less progress than the state growth standard.

Students are making substantially less progress than the state growth standard.


Standard Error

A measure of the uncertainty All measures of student learning contain error.

In the EVAAS teacher value-added report, the size of the standard error is influenced by N size (size of the student group). Missing scores.


Year 1 Estimate = 1.5

4

3

2

1

0

-1

-2

4.0

-1.0

True value lies somewhere within the range of the standard error

Estimate is Most likely value

Y1 Y2 Y3Multi-yr.

Ave.


Teacher Value-Added Report

Note: Battelle for Kids is utilizing visual representations of copyrighted EVAAS® Web reporting software from SAS in this presentation for instructional purposes.



Growth Index Since the size of the standard error (degree of

certainty) will vary across teachers, their estimated gain must be standardized to include both the estimate and the degree of certainty (standard error).

Divides a teacher’s estimated gain by the associated standard error.


The Teacher Value-Added Report


The Teacher Progress Table


Levels of Teacher Value-Added Effects

Most Effective

Teacher's index: 2 or greaterStudents are making substantially more progress than the state growth standard.

Above Average

Teacher's index: equal to or greater than 1, but less than 2 Students are making more progress than the state growth standard.

AverageTeacher's index: equal to or greater than -1, but less than 1Students are making about the same amount of progress as the state growth standard.

Approaching Average

Teacher's index: equal to or greater than -2, but less than -1 Students are making less progress than the state growth standard.

Least Effective

Teacher's index: less than -2Students are making substantially less progress than the state growth standard.


Foundational Statistics

GR Ohio’s Academic Content Standards Mathematics: Statistical Methods

10Interpret the relationship between two variables using multiple graphical displays and statistical measures (e.g., box-and-whisker plots and measures of center and spread).

8Describe how the relative size of a sample compared to the target population affects the validity of predictions.Explain the mean’s sensitivity to extremes…

7…describe how the inclusion and exclusion of outliers affect those measures.

6Understand the different information provided by measures of center (mean, mode, median) and measures of spread (range).

1Describe the likelihood of simple events as possible/impossible and more likely/less likely.


Key Considerations: Systemic Programs, Delivery Models, Structures,

Services, etc. Professional Learning Curriculum Alignment

What I teach Assessment

How I measure/monitor learning along the way Instruction

How I teach

Using Value-Added to Inform Practice


Combining Measures to Inform Practice All measures of student learning contain

error.

No single measure can capture the complexity of learning and teaching.

There’s an important distinction between a flawed measure and a flawed assumption based upon a single measure.

@BattelleforKids

facebook.com/battelleforkidsorg

youtube.com/battelleforkids

BattelleforKids.org

Questions?

Thank You!

teacher value-added reports state board of education january 14, 2014 jamie meade managing director,...

Documents