a model description by benjamin ditkowsky, ph.d. student growth models for principal and student...
TRANSCRIPT
A Model Description ByBenjamin Ditkowsky, Ph.D.
Student Growth Models for
Principal and Student Evaluation
It’s Not Always Easy To See What Our Students Can And Can Not Do
In Addition, Sometimes Some Things Just In Addition, Sometimes Some Things Just Don’t MatchDon’t Match
Can We Just Use The Same Test At Two Points In Time (Use Gain Scores)?
There are some problems with gain scores All tests are made up of a true score and
measurement error xi = xiTRUE +
Error is always present, but we don’t know how much error is present
When we subtract one score from another (i.e., a gain score) error in the final score is greater than in either of the tests from which the error originates
xgain = x2TRUE– x1TRUE + + (true differences are subtracted but error propagates)
Error! There Are Many Reasons Other Than The Test That Student Scores Vary From One Test To Another
Complex Solutions Exist, But
Do Complex Solutions improve teaching and learning?
Do Complex Solutions maintain a focus on the connection on standards?
Do Complex Solutions applied to our data violate basic statistical assumptions?
Do Complex Solutions focus on details or big ideas?
The overly complex statistical models have not been shown to be substantially more effective than less complex models
with complexity comes other concerns
Accountability Should Be
Trustworthy For a solution to be trustworthy it should meet all of the
assumptions upon which it is based
Useable For a solution to be useable, it should help teachers and
principals adjust instruction and intervention in real time
Accessible For a solution to be accessible it should be transparent
We Will Use Student Data For Evaluation
We Have Lots Of Data
Complex solutions have complex algorithms to estimate missing information.
For most students… we have enough information we can infer the rest.
Although, we don’t necessarily have the same data for all students…
Complex Statistical Analysis Is Unnecessary
Given sufficient evidence, complex analysis is unnecessary
looks like a duck
walks like a duck
swims like a duck
quacks like a duck
it probably is a duck
If it …
We Know What We Expect
Cut Scores are Based on Expected Performance
Test performance is categorized as Exceeding Standards, Meeting Standards, Below Standards, Academic Warning
Progress is defined by our assignment of the value or worth of improvement in the categorical performance of test scores from one time to the next
We Use Cut Scores To Categorize Level of Achievement, and Progress
Cut Scores Separate Performance Levels
ISAT Scores are broken down into performance designations (Academic Warning, Below Standards, Meets Standards, Exceeds Standards)
Typical Growth in Language Development on ACCESS has been established (W.I.D.A. Consortium, 2009)is calculated based on entry performance level and change in composite score (typical low, and high range growth)
R-CBM Scores reliably predict state test scores (Silberglitt & Hintze, 2005), cut scores predicting outcomes have been established by aimsweb and within Illinois (i.e., Below Basic, Questionable, Proficient, confidently proficient)
M-CAP scores predict State Test scores, and typical performance is established across the township
Benchmark assessments can divide score into performance categories
We Are Explicit About Our Values
Exceptional Growth (2) as well as maintenance of exceptional performance is highly valued, thus any score-pair that moves up two or more categories or ending in the highest performance category is worth two points.
Proficient Growth (1) is defined as any score-pair that moves up one category or is maintained i.e., scores growth at a rate commensurate with the increasing expectations of the Meeting Expectations category.
Inadequate Growth (-1) is defined as a score-pair in which the performance category at time 1 is higher than the performance category at time 2, or growth from time 1 to time 2 is not sufficient to move a student from the below expectations category into the meeting expectations category.
Unsatisfactory Growth (-2) is defined as a score-pair in which the performance category drops by two categories from time 1 to time 2 or ends in the Academic Warning Category.
Growth is defined as a change in performance category from Time 1 to Time 2
A Value Table Weights Growth
A. Warning Below Meets Exceeds
Academic Warning -2 1 2 2
Below Expectations -2 0 1 2
Meets Expectations -2 -1 1 2
Exceeds Expectations -2 -2 -1 2
Time 1 PerformanceTime 2 Performance Level
Miguel Is A 2nd Grade Student
The Data we have for him include:
Type I ACCESS from grade K and 1 Aimsweb - Fall and Winter
RCBM MCAP (not in example)
Type II Vocabulary Matching Fall and
Winter Avenues Pre-Post Assessments for
ELLs
Type III Pre-Post Classroom Assessments
Name Grade K Grade Level K 1 Fall Winter Fall Winter Pre Post
Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3
R CBM Vocabulary Matching AvenuesACCESS Composite
Name Grade K Growth ValueGrade Level K 1 Pattern Fall Winter Fall Winter Pre Post
Miguel 2 1.8 209 297 Proficient 1 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 Excellent 2 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 Needs to Improve 0 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 Proficient 1 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 Unsatisfactory -1 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3
R CBM Vocabulary Matching AvenuesACCESS Composite Each ACCESS pair is assigned a growth value based on the
comparison of scores and the expectations for growth
(WIDA, March 2009)
Unsatisfactory
Needs Improvement
Proficient growth
Excellent growth
Miguel is in second grade. • Baseline: In K his language proficiency level
was 1.8• Expectations: The low to high range for growth
was 44 to 90• Scores: 297 – 209 = 88
His gain was 88, less than 90 • Category designation: His growth level is
considered in the proficient range
Miguel Is A 2nd Grade Student
Name Grade K Grade Level K 1 Fall Winter Fall Winter Pre Post
Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3
R CBM Vocabulary Matching AvenuesACCESS Composite
Name Grade K Growth ValueGrade Level K 1 Fall Winter Pattern Fall Winter Pre Post
Miguel 2 1.8 209 297 15 (BB) 66 (PR) Excellent 2 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) Needs to Improve 0 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) Unsatisfactory -1 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) Excellent 2 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) Proficient 1 1 (B) 8 (M) 1 3
R CBM Vocabulary Matching AvenuesACCESS Composite
In Winter, Miguel’s performance was in the
proficient range
Fall performance
was Below Basic Expectations
Grade Measure Below Basic Proficient Below Basic Proficient Below Basic Proficient
2 R-CBM 30 45 55 65 70 90
Fall Winter Spring
Miguel is in second grade. • Baseline: In Fall his score 15 WRC indicated Below Basic Performance• Expectations: Winter Proficient Score is 65 WRC• Growth: In Winter, Miguel scored in the proficient range, his movement up two
categories (Below to Proficient) is considered excellent growth• Category designation: His growth level is considered in the excellent range
MeasuredEffects.com, 2010 ISAT Cut Scores
Miguel Is A 2nd Grade Student
Name Grade K Grade Level K 1 Fall Winter Fall Winter Pre Post
Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3
R CBM Vocabulary Matching AvenuesACCESS Composite
Name Grade K Growth ValueGrade Level K 1 Fall Winter Fall Winter Pattern Pre Post
Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) Proficient 1 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) Excellent -1 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) Proficient 2 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) Excellent 2 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) Proficient 1 1 3
R CBM Vocabulary Matching AvenuesACCESS CompositeRelatively low scores are in the proficient range in the fall, Miguel’s score of 2 is
considered proficient
In Winter, expectations for VM are higher, and
Miguel’s score increased sufficiently to remain on
target.
VMGrade 5 10 25 50 75 90 5 10 25 50 75 90
2 0 0 0 2 5 8 1 2 4 7 10 13Warning Exceeds Warning Exceeds
Winter
Robust Percentile Rank Robust Percentile Rank
Below Meets Below Meets
FALL
Local Normative Values
Miguel is in second grade. • Baseline: In Fall his score 2 WRC indicated his performance was typical in the Meets
category• Expectations: Winter Proficient range is from 7 to 10• Growth: In Winter, Miguel scored in the proficient range, his performance indicated
that his growth was consistent with expectations for his grade level (Meets to Meets)• Category designation: His growth level is considered in the proficient range
Miguel Is A 2nd Grade Student
Name Grade K Grade Level K 1 Fall Winter Fall Winter Pre Post
Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3
R CBM Vocabulary Matching AvenuesACCESS Composite
Name Grade K Growth ValueGrade Level K 1 Fall Winter Fall Winter Pre Post Pattern
Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2 Proficient 1Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5 Excellent -1Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5 Excellent 2Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4 Excellent 2Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3 Excellent 1
R CBM Vocabulary Matching AvenuesACCESS Composite
The Avenue’s pre test is broken into 6 levels designated in 3 levels (Beginning,
Intermediate and Advanced)
Miguel is in second grade. • Baseline: In Fall his proficiency level was 1 indicating early beginning language
proficiency• Expectations: Winter Intermediate language range is from 3 to 4• Growth: In Winter, Miguel’s language level increased from level 1 to 2, though not up
to the Intermediate range, his scores demonstrated growth• Category designation: His growth level is considered in the proficient range
Miguel Is A 2nd Grade Student
A Demonstration Of Proficient Growth
• Each score, for each second grade student in the targeted ELL subgroup has been examined and categorized based on Cut Scores.
• Each available score-pair has been reviewed and growth has been categorized and weighted.
• Individual ratings were calculated
• The overall group rating was calculated
Pattern Value Pattern Value Pattern Value PatternMiguel 2 Proficient 1 Excellent 2 Proficient 1 Proficient 1 1 ProficientAugustine 2 Excellent 2 Needs to Improve 0 Excellent -1 Excellent -1 -1 UnsatisfactoryIsabelle 2 Needs to Improve 0 Unsatisfactory -1 Proficient 2 Excellent 2 1 ProficientFariha 2 Proficient 1 Excellent 2 Excellent 2 Excellent 2 2 ExcellentNabiha 2 Unsatisfactory -1 Proficient 1 Proficient 1 Excellent 1 1 Proficient
Group Rating 1 Proficient
Name Grade Individual Growth
ACCESS Composite R CBM Vocabulary Matching AvenuesGrowth
j
mMiguel = m[1,2,1,1] = 1
Rating = m[1, -1, 1, 2, 1] = 1 1 Proficient Growth1 Proficient Growth
Alternate Illustration: Why Use More Than Two Administrations Of The Same Test From One Source
j
It is possible that different tests indicate different patterns, without changing the overall rating
The differences in patterns by test may be diagnostically important, but insufficient for high stakes evaluative purposes
English Language proficiency is growing at a rate above what
is expected
Automaticity with basic skills in
Reading may need to improve
Automaticity with basic skills
in Math may be on track
Instructional Vocabulary may be insufficient for continued
growth in grade level material
MCAP
Pattern Value Pattern Value Pattern Value Pattern Value PatternMiguel 2 Proficient 1 Excellent 2 Needs to Improve 0 Proficient 1 1 ProficientAugustine 2 Excellent 2 Needs to Improve 0 Needs to Improve 0 Excellent -1 Unsatisfactory -1 0 UnsatisfactoryIsabelle 2 Needs to Improve 0 Unsatisfactory -1 Excellent 2 Excellent 2 Needs to Improve 0 0 ProficientFariha 2 Excellent 2 Needs to Improve 0 Proficient 1 Needs to Improve 0 Proficient 1 1 ExcellentNabiha 2 Excellent 2 Unsatisfactory -1 Needs to Improve 0 Proficient 1 Proficient 1 1 Proficient
Excellent 2 Needs to Improve 0 Proficient 1 Needs to Improve 0 Proficient 1 1 Proficient
Name Grade Individual Growth
ACCESS Composite R CBM Vocabulary Matching AvenuesGrowth
Growth English Language Proficiency is increasing at an adequate rate
While there may be some areas where progress is worth further
investigation, for the purposes of evaluating and categorizing; overall, across measures,
academic growth is occurring at an acceptable rate.
Timmy Is A 4th Grade Student
The data we might have for him include:
Type I ISAT from grade 3 and 4 Aimsweb - Fall and Winter
RCBM MCAP
Type II Vocabulary Matching
Type III Pre-Post Classroom Assessments
What Data Might We Use For Him?
Gertrude Is A 7th Grade Student
The data we might have for her include:
Type I ISAT from grade 3 and 4 Aimsweb - Fall and Winter
RCBM MCAP
Type II Vocabulary Matching Prentice Hall Benchmark Assessments
Type III Pre-Post Classroom Assessments
What Data Might We Use For Her?
Setting Student Data Goals For Principal Evaluation
By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students at administered at two points in time school wide, students at school school namename will demonstrate an increase in the proportion making will demonstrate an increase in the proportion making adequate progress from 35% to 40%. adequate progress from 35% to 40%.
Goals can be set to increase the amount of progress made
Setting Student Data Goals For Principal Evaluation
By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students identified administered at two points in time school wide, students identified as as define cohortdefine cohort at at school nameschool name will demonstrate an increase in the will demonstrate an increase in the proportion making adequate progress from 68% to 80%.proportion making adequate progress from 68% to 80%.
Goals can be set to increase the proportion of students making adequate progress made for a particular subgroup
Setting Student Data Goals For Principal Evaluation
By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students identified administered at two points in time school wide, students identified as as define cohortdefine cohort at at school nameschool name will demonstrate a decrease in the will demonstrate a decrease in the proportion making unsatisfactory progress from 8% to 4%.proportion making unsatisfactory progress from 8% to 4%.
Goals can be set to decrease the proportion of students not making adequate progress for a particular subgroup
Setting Student Data Goals For Principal Evaluation
By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students at administered at two points in time school wide, students at school school namename will demonstrate will demonstrate proficient or excellentproficient or excellent growth as defined by growth as defined by the convergence and magnitude of data classified with district the convergence and magnitude of data classified with district defined value tables.defined value tables.
Goals can be set to achieve an overall rating for the demonstration of student growth