School Improvement, Short Cycle Assessments and Educator Evaluation
OrlandoJune 22, 2011
Allan Odden and Anthony Milanowski
Strategic Management of Human Capital (SMHC)
University of Wisconsin-Madison
Overview of Presentation
1. Prime challenge is to improve student performance
2. Key strategy to attain that goal (on which I focus today): talent and human capital management
3. Support tactic for talent management – multiple measures of effectiveness used in new teacher evaluation systems
2
Improving Student Performance
• CPRE research and Lawrence O. Picus and Associates research from school finance adequacy studies: Odden (2009) & Odden & Archibald (2009)
• Research by others – Ed Trust, Karen Chenoweth (2007), Supowitz, etc.
• Schools from urban, suburban and rural communities • Many schools and districts with high concentrations of
children from low income and minority backgrounds• Finalists for the Broad Prize in Urban Education• = Ten Strategies for Improving Performance
Ten Strategies to Improve Performance
1. Initial data analysis, largely analyzing state accountability tests
2. Set high ambitious goals – double student performance, 90% to advanced standards
3. Adopt new curriculum materials and over time a systemic view of effective instructional practices that all teachers are expected to implement
4. Implement data based decision making with benchmark and short cycle assessments – e.g. Renaissance Learning STAR Enterprise
Ten Strategies to Improve Performance
5. Invest in comprehensive, ongoing professional development, including instructional coaches in all schools
6. Use school time more effectively – protected core subject times for reading and math – collaborative time for teacher teams working in Professional Learning Communities
7. Multiple extra-help strategies for struggling students – tutoring, extended day, summer
8. Widespread distributed instructional leadership
Ten Strategies to Improve Performance
9. Reflect “best practices” and incorporate research knowledge – not doing your own thing
10. Be serious about talent – finding it, developing it, determining effective from ineffective teachers and principal; promoting, highly compensating and retaining only teachers and principals based on measures of effectiveness
Human Capital Management
• Obama and Duncan administration has made improving teacher and principal talent and their effectiveness central to education reform
• Goal: put an effective teacher into every classroom and an effective principal into every school
• To implement these practices and manage teachers (and principals) around them, develop multiple measures of teacher effectiveness (long-hand for new teacher evaluation systems)
• Scores of states and districts working on this issue• These issues also central to ESEA reauthorization• The question is not whether teacher evaluation will change but
how it will be changed
7
Core Elements of the Strategy
• Multiple Measures of Teaching Effectiveness1. Measures of instructional practice – several systems
2. Measures of pedagogical content knowledge
3. Student perceptions of the academic environment
4. Indicators of impact on student learning
All this is now mandated by Illinois law
• Use of those measures:a) In new evaluation systems, for teachers and principals
b) For tenure
c) For distributing and placing effective teachers
d) For dismissing ineffective teachers
e) For compensating teachers
8
Teacher Evaluation
Two major pieces of the evaluation:1. Measure of instructional practice – Danielson Framework,
INTASC, Connecticut BEST system, CLASS, PACT, National Board, the new North Carolina system – see Milanowski, Heneman, Kimball, Review of Teaching Performance Assessments for Use in Human Capital Management, 2009 at www.smhc-cpre.org and go to resources
2. Measure of impact on student learning:a. The only model at the present time is value added using end of year
state summative testsb. One new proposal is to use interim-short cycle (every 4-6 weeks)
assessment data, aligned to state content standards, that show student/classroom growth relative to a normed (national or state?) growth trajectory
9
Other National Efforts
• Measuring Effective Teaching project of the Gates Foundation:– Multiple value added measures– Several teacher rubrics, with video tool to
replace direct observations– Student survey – Ron Ferguson
10
Measuring Educator Performance
The indicators of impact on student learning, must devolve from tests that:
1. Are valid and Reliable
2. Are instructionally sensitive and instructionally useful (linked to state content standards and provide data to teachers about how to improve instructional practice)
3. Provide stable results, which mean they should be given multiple times a year (every 4-6 weeks)
Many state accountability tests fall short of these psychometric standards
14
Measuring Educator Performance
Be very helpful if the data system can be used:
• By teachers to guide their instructional practice• To roll up the individual data to the classroom to
indicate teacher impact on student learning gains• To roll up the individual data to the grade level and/or
the school level to indicate impact of school and school leadership on student learning gains
15
Final Contextual Comment
• All these systems must be embedded within a framework of ongoing educator development
AND
• During these tight fiscal times, funds for professional development should NOT be cut
16
Multiple Measures of Teaching Performance for Accountability & Development
• Standard Prescription: Instructional practice measure (e.g., teacher
evaluation ratings) + Gain, growth, or value-added based on state standards-based assessments
• But:– Practice ratings and assessment gain, growth, or
value-added don’t measure the same thing; measurement error sources are different and don’t cancel
– Gain, growth, or value-added on state assessments are of limited use for teacher development
Advantages of Adding Short-cycle Assessments to the Mix
1. For teacher development: because such assessments are frequent, teachers get feedback that they can use to adjust instruction before the state test
– Teachers can see if student achievement is improving, and if assessments are linked to state proficiency levels, whether students are on track to proficiency
2. For teacher accountability: – More data points allow estimation of a growth curve – The growth curve represents learning within a single school
year; no summer to confuse attribution– The slope of the average growth curve or average difference
between predicted end points provides another indicator of teaching effectiveness
– Combining with growth, gain, or value-added based on state assessments provides multiple measures of productivity
– If linked to state assessments, can predict school year proficiency growth
Issues in Combining Practice & Student Achievement Measures
• Models: Report Card, Compensatory, Conjoint
• When Combining Need to Address:– Different Distributions, Scales and Reference
Points– Weighting in Compensatory Models
• Equal• Policy• Proportional to reliability
Report Card Model
22
Performance Domain
Performance Dimensions
Score Levels Requirement for Being Considered Effective
Instructional Practice
Planning & Assessment Classroom Climate Instruction
1-41-41-4
Rating of 3 or higher on all dimensions
Professionalism CooperationAttendanceDevelopment
1-41-4 1-4
Rating of 3 or higher on all dimensions
Student Growth, Gain, or VA on State Assessments
MathReading/ELAOther Tested Subjects
Percentiles in state/district distribution for each subject
Being in the 3rd Quintile or Higher for All Tested Subjects
Student Growth on Short Cycle Assessment
MathReading
Avg. Growth Curve Translated into Predicted State Test Scale Score Change
Predicted Gain Over Year Sufficient to Bring Student from Middle of “Basic” Range to “Proficient”
Putting Practice Ratings andStudent Achievement on the Same Scale
Emerging Practice: Rescale growth, gain or value-added measure to match the practice rating scale – Standardize and set cut-off points in units of
standard error, standard deviation or percentiles
Category In S.E. Units PercentilesDistinguished (4) >1.5 S.E. Above Mean 70th +
Proficient (3) +/- 1.5 S.E. Around Mean 30th to 69th
Basic (2) 1.51 - 2 S.E. Below Mean 15th to 29th
Unsatisfactory (1) > 2 S.E. Below Mean Below 15th
Compensatory (Weighted Average) Model for Combining Performance Measures
Dimension Rating Weight Product
Growth, Gain, Value-Added on State Test
2 25% 0.50
Growth as Measured by Short-Cycle Assessment
3 25% 0.75
Practice Evaluation 4 50% 2.00
3.25
1.0-1.75 = Unsatisfactory, 1.76-2.75 = Basic, 2.76-3.75 = Proficient, 3.76 += Distinguished
25
Conjoint Model for Combining 2 Measures
Student Outcome Rating
TeachingPractice
1 2 3 4
4 = Advanced
2 2 3 4
3 = Proficient
2 2 3 4
2 = Basic 1 2 2 3
1 =Unsatis- factory
1 1 1 2
26
Conjoint Model for Combining 3 Measures
To Get aSummary Rating of
Need Scores of at Least:
4 4 on two measures and 3 on the other
3
2 on the practice measure and 4 on both the student achievement measures- or - 3 on the practice measure and 3 on at least one of the student achievement measures
2 2 on the practice measure and 2 on either of the student achievement measures
1 1 on the practice measure and 1 on either student achievement measure
27
Impact on Student Learning
• The conversation is about “multiple indicators” for this category
BUT• Few if any places actually have viable multiple
indicators• The prime and in most cases only indicator here
is a “value added measure derived from state summative, accountability tests”
29
Impact on Student Learning
• Most teachers do not like value-added measures using end of year state summative tests; don’t understand them; don’t like state tests
• So what could be actual and practical additional indicators, indicators that could augment these value added statistics that derive from state summative tests (which whatever our viewpoint will probably not go away)
30
Impact on Student Learning
• Interim, short cycle assessments, that are given multiple times during the year
• Interim short-cycle assessments (STAR is one example) are used to help teachers improve instruction and also can be used to show student and classroom growth
• The only new, viable, specific idea in this area that is now on the table, and gives comparable evidence across teachers
31
Several Additional Indicators• Background points:
– STAR Reading and Math cover grades K-12 so cover classrooms above the standard grade 3-8 and 11
– Administered in computer-based format so provide immediate feedback to teachers for use in instructional improvement and change
– Vertically aligned scales so can compare scores across months and years
– Following charts derive from individual student data
32
Chart 1• Interim assessments given monthly
• Student data aggregated to classroom
• Red squares are progress line for similar classes of students in a state (or nation)
• Green triangles are actual class progress
• Yellow star is state proficiency level
• Shows growth during the months of just the academic year
34
Chart 1• Modest student learning when the class had a
substitute teacher• Growth happened when regular teacher returned• Actual class growth (green triangles) was much
greater than the reference norm (red squares)• In value added terms, the class would have a
high value added – performance growth was above the average (the red square trend line) for this typical classroom
35
Chart 1 How to use these data:
6. Compare growth of this class to other classes:a. In the same school
b. In the same district
c. With similar demographics
d. In the same state
e. Across the nation
f. To classes in schools with similar demographics
Many different ways to use the data in such a Chart
36
Chart 2
• Interim assessments given monthly
• Student data aggregated to school level
• Red squares are progress line for schools with similar students in a state (or nation)
• Green triangles are actual school progress
• Yellow star is state proficiency level
• Shows growth during the months of just the academic year
38
Chart 2
• Rolls the classroom data up to the school level
• Could also roll up student data across classes for each grade
• Multiple ways to create an indicator
39
Chart 2 How to use these data:
1. Shows school performed above the state proficiency level
2. Shows school performed above similar schools
3. So compare end of year score on interim assessments with end of year score on state summative test – do both show exceeded proficiency?
4. Compute “standard deviation” of change – fall to spring and COMPARE to “standard deviation” of change on state summative test
5. Compare end-of-year interim assessment score to end-of-year state proficiency score, in terms of standard deviation above proficiency level, or standard deviation of growth over the year
40
Chart 2
• Compare to other schools in the district
• Compare to other schools in the state
• Compare to other schools in the nation
• Compare to schools with similar demographics
• Compare value added or growth scores on interim assessments to that on state summative assessments
41
Chart 2
• When data are rolled up to the school level, they provide additional indicators for:– Those education systems, like Hillsborough
(FL), which are using school wide gains for teachers of non-tested subjects
42
Chart 3
• Indicates whether classroom (school) is low performance level but high or low growth OR high performance level and high or low growth
• Could use simply to give points if high growth, or negative points of both low performance level and low growth (indicates a real performance issue)
44
Final Comments• Interim short cycle assessments can be used
to provide additional indicators of teacher (school) impact on learning growth
• The data supplement what is shown by value added with state summative tests
• Thus such data reduce the weight given to such indicators
• And these data derive from a system designed to help teachers be better at teaching
45
Allan OddenUniversity of Wisconsin-