Cut Scores, Student Growth and College/Career Readiness: 2011-2012 Data Dialogues
Stan MastersLenawee ISD
January 17, 2012
• Data Dialogues around “hot topics” for 2011-2012 are occurring in Lenawee County.
• Using data stored in DataDirector provides ways to create reports that can be used in these dialogues.
• Participants will learn about using data to have conversations that deepen understanding of these "hot topics."
• Participants should bring their username and password for DataDirector.
Goals For Today
• Describe how the new cut scores will impact the MEAP/MME– Refine Data from 10-11 MEAP
• Describe how a growth model operates – Create an Assessment with Calculation for Growth
• Describe how multiple measures help define “College and Career Readiness” – Open an EXPLORE, PLAN, or Elementary Report
in Excel and color code College Readiness
Norms for Our Work
• Participate actively• Actively listen• Seek application• Press for clarification• Honor time agreements and confidentiality• Keep ‘side bars’ to a minimum and on topic• Take care of adult learning needs
WE MUST UTILIZE AN INQUIRY APPROACH
TO DATA ANALYSIS
WE MUST USE MULTIPLE SOURCES
OF DATA
We need a data warehouse
for our 21st century schools
WE MUST FOCUS ON DATA TO INCREASE STUDENT ACHIEVEMENT
Talking Points for the Purpose of Implementing
a Data Warehouse in Lenawee Schools
Dialogue About
Meaning
Systematic
Multiple Sources
Analysis
WE MUST UTILIZE AN INQUIRY APPROACH
TO DATA ANALYSIS
WE MUST USE MULTIPLE SOURCES
OF DATA
We need a data warehouse for our 21st century schools
WE MUST FOCUS ON DATA TO INCREASE STUDENT
ACHIEVEMENTData-Driven
CultureProcess
Data-DrivenDecision-Making
Key Characteristics of the Objectives of the Data Warehouse
Taken from the Michigan School Improvement Framework
LISD Data DirectorStan MastersCoordinator of
Instructional Data Services
• “Data out”• Creation and analysis of
data-driven reports
Mike HusbandData Entry Specialist
• “Data in”• Organization of data to
add to the warehouse
FERPA/HIPAA Pre-Test
To be considered an “education record,” information must be maintained in the student’s cumulative or permanent folder.
• False, because any record that has a student name is an educational record.
FERPA/HIPAA Pre-Test
You are in charge of a staff meeting to study student achievement on school improvement goals. As part of your meeting, you are showing a report to the entire staff that shows student scores on a common local assessment. The report shows the student names. In addition, you have given them a paper copy of the report.
It is a violation of FERPA to display the results of the assessment to the entire staff.
The exception would be a group of teachers working on a specific student strategies, as they are a specific population that then has a “legitimate educational interest” in the information.
Login
• https://www.achievedata.com/lisd• Username:
– 3-letter school abbreviation +(first 4 letters of last name)(first letter of first name)
• Password: – For new users: 123456dd (you will create a new password
after logging in). – For existing users: use your existing password.– I can reset passwords to 123456dd
Data Driven Dialogue
di·a·logue or di·a·log n. Abbr. dial. 1. A conversation between two or
more people. 2. 2. An exchange of ideas or
opinions: achieving constructive dialogue with all parties present. --di·a·logue v.
Deb Clancy, Washtenaw ISD, 2008
Ways of Talking
Normsof
Collaboration
Dialogue Discussion
Outcome:Deep
Understanding
Outcome:DecisionsThat Stick
Culture of Collaboration
Conversation
Deliberation
The Center for Adaptive Schools www.adaptiveschools.com
What do you need to monitor?
• Progress on school improvement goals• Tracking individual student progress• Prioritize key indicators• Support teaching and learning
Source: “Developing a Monitoring Plan”. Maryland Department of Education. Accessed May 25, 2010 from http://mdk12.org/data/progress/developing.html
How do you develop a monitoring plan?
• Identify specific learning indicators• Create data collection templates• Schedule assessment calendar
– collaborative collection and analysis
Source: “Developing a Monitoring Plan”. Maryland Department of Education. Accessed May 25, 2010 from http://mdk12.org/data/progress/developing.html
Video Source: Reeves, D. (2009). “Planning for the New Year”. Accessed May 25, 2010 from http://www.leadandlearn.com/webinars
Old Cut Scores
• Scaled Scores represent the stable score on the assessment that is reported for each student.
• Floor of “Meets” performance level was calculated as (Grade level X 100)
3rd grade “Meets” = 3 X 100 = 300
Grade Subject Correct Total Percent
3rd Reading 13 30 43%
3rd Math 21 37 57%
Fall 2008 Passing Cut Scores Grades 3rd-5th
Grade Subject Correct Total Percent
4th Reading 13 30 43%
4th Math 20 48 42%
Grade Subject Correct Total Percent
5th Reading 14 30 47%
5th Math 21 49 43%
5th Science 24 51 47%
Source: Presentation by Terri Portice, June 29, 2009
Fall 2008 Passing Cut Scores Grades 6th – 9th
Grade Subject Correct Total Percent
6th Reading 14 30 47%
6th Math 20 47 43%
6th Social Studies 24 46 52%
Grade Subject Correct Total Percent
7th Reading 15 30 50%
7th Math 22 50 44%
Grade Subject Correct Total Percent
8th Reading 17 30 57%
8th Math 13 38 34%
8th Science 24 58 41%
Grade Subject Correct Total Percent
9th Social Studies 23 46 50%
Source: Presentation by Terri Portice, June 29, 2009
2009 MME Percent Correct Needed to “Pass”
Content Percent
Reading 63%
Writing 66%
Math 56%
Science 54%
Social Studies 44%
Source: Personal E-mail Correspondence with Ernie Bauer, Oakland Schools, July 15, 2009
Change in Cut Scores for MEAP and MME
Comparing “Old” vs “New”Using Fall 2010 and Spring 2011 data
Michigan and Lenawee
• Open MEAP or MME Percent Proficient Report• Click on Percentage of Students Proficient• Make This a Report• Refine the data to Reflect the New Cut Scores
Predictions Observations Inferences I predict . . . I can count . . . I believe that the
data suggests . . . because, …
9591
8084 85
78
52
8784 85 84
7982
63
78 78
61
75 7478
3537
3936 36
29 28
63 6466
64
56 5754
1714
2628
33
42
0
10
20
30
40
50
60
70
80
90
100
3rd 4th 5th 6th 7th 8th 11th 3rd 4th 5th 6th 7th 8th 11th 5th 8th 11th 6th 9th 11th
Math Reading Science Social Studies
Per
cen
t P
rofi
cien
tMEAP & MME 2010-11 Old & New Cut Scores
Michigan Old Michigan New
9693
83 8486
80
46
9187
9086
82 83
62
8583
61
79 80 80
2832 33
2730
25
18
70
65
70
63
5654
50
17
12
22 23
3640
0
10
20
30
40
50
60
70
80
90
100
3rd 4th 5th 6th 7th 8th 11th 3rd 4th 5th 6th 7th 8th 11th 5th 8th 11th 6th 9th 11th
Math Reading Science Social Studies
Per
cent
Pro
ficie
ntMEAP & MME 2010-11 Old & New Cut Scores
Lenawee County Old Lenawee County New
37
Michigan School Reform Law
• Conduct annual educator evaluations
• Include measures of student growth as a significant factor
• Locally determine the details of the educator evaluations, the consequences, and the timeline for implementation.
Key Characteristics of Growth Models• Data must align with
agreed-upon content standards
• Data must measure a broad range of skills
• Data must document year-to-year growth on a single scale
Measuring Student Growth: A Guide to informed decision making. (2007). Center for Public Education.Using Student Progress to Evaluate Teachers: A Primer on Value-Added Models. (2005). Education Testing Service.
Growth Models
• Improvement Model• Performance Index• Simple Growth• Growth to Proficiency• Value-Added
Improvement ModelCompares one cohort of students with another cohort in same grade/course
• Benefits– Easy to implement– Simple to communicate
• Disadvantages– Does not track individual
student progress– Does not take into
account other factors that may have promoted/inhibited growth
4th Graders from 2010-11 to 2011-12
Performance IndexCombines multiple data sets into a single scale
• Benefits– Recognizes changes in all
achievement levels– Uses multiple measures– Can lead to
improvement for all students, not just “bubble” students
• Disadvantages– Does not track individual
student progress– Do not capture change in
each achievement level– May be desirable to use
more achievement levels
Grade-Point Average
Simple GrowthFollows same cohort of students
• Benefits– Uses scaled scores from
one year to the next– Documents changes in
individual students
• Disadvantages– Includes only the
students present for both years
– Need to determine how much growth is enough
MEAP Scale Scores
Growth to ProficiencyDesigned to show if students are “on-track”
to meet standards
• Benefits– Provides more data
points toward goal– Recognizes gains even if
students are not proficient
– Focus on all students, not just “bubble” students
• Disadvantages– Targets must be
determined by outside agencies
– Benchmark points must be agreed upon
Cut Scores for Proficiency
Value-AddedPast performance used to predict future scores
• Benefits– Measures student
performance over time– Documents the impact
of instructional resource, program, or school process on the change
• Disadvantages– Complex statistics– Isolates student
demographics that may impact performance
EXPLORE to PLAN to ACT
Examples of Growth AssessmentsSource: Britton-Deerfield Teacher-Evaluation Committee, 2011
• Local– Classroom tests, performance
assessments, IEP goals, portfolio exhibits
• State– MEAP, MME (ACT), MI-Access
• National– DIBELS, STAR, NWEA, EXPLORE,
PLAN
‘data from multiple sources ’
Key Characteristics of Growth Models• Data must align with agreed-upon content standards
– Identify significant standards for growth– Align assessment and instructional plans
• Data must measure a broad range of skills– Develop assessment instruments (test blueprints,
performance rubrics, and scoring guides)– Construct assessment calendar (beginning to end of year)
• Data must document year-to-year growth on a single scale– Determine initial threshold scores for determining growth
Pre-Test and Post-test
• If you are using your pretest for comparison to post-tests, then you will want very similar questions on both.
• The best pre-tests cover exactly the same objectives as the test
• Make sure the pre-test conditions are as similar to the mode used in post-test
Example of teacher using a pre-test/post-test model
Sand Creek High SchoolPre-Test and Post-Test
Student Name
Life Skills Pre-test
Life Skills Post-test
Life Skills Growth
Life Skills Writing Pretest
Life Skills Writing Post
test
WritingGrowth
95 95 0 55 64 9
85 90 5 38 92 54
70 75 5 51 65 14
65 76 11 32 116 84
60 65 5 50 54 4
65 70 5 70 80 10
65 50 -15 103 130 27
Procedures• Administered before and after instruction• Look at the scores of individual students
to determine how many had higher post-test scores (Simple Growth Model)
• Compare the percentage to the threshold agreed upon by school/district
• Calculate the mean pre-test score and compare that with the mean post-test score (Simplified Value-Added Growth Model)
Source: Measurement Issues Inherent in Educator Evaluation, Presentation by the Michigan Assessment Consortium to the OEAA Educator Evaluation Best Practices Conference, April 15, 2011.
ExamplesIndividual
• A teacher gives 4 pre-post tests during the year.
• For each sequence, the teacher calculates an average change from pre-test to post-test.
• The teacher compares the number of students whose scores changed in various amounts.
Common• A district develops 2 tests to
be given pre-post during the year for specific content/grade levels.
• For each sequence, the district calculates an average change from pre-test to post-test.
• The district compares the number of students whose scores changed in various amounts.
Source: Measurement Issues Inherent in Educator Evaluation, Presentation by the Michigan Assessment Consortium to the OEAA Educator Evaluation Best Practices Conference, April 15, 2011.
Onsted High Considers Thresholds for Student Growth and Achievement
Which of our 2010-2011 Freshmen met or exceeded NWEA Common Core Target growth rate or met or exceeded 60th percentile in Reading or Math?
Change in Reading Score Change in Math Score NWEA Spring 2011 NWEA Spring 2011
Spr 2010 to Spr 2011 Spr 2010 to Spr 2011 Reading Test Percentile Math Test Percentile
8 -13 43 31
4 6 69 55
10 3 62 24
3 -17 55 23
0 5 79 63
-1 -7 62 55
13 -15 48 16
Onsted Middle School,NWEA Growth, by Teacher, by Class PeriodStudent Name
Teacher Name
Class Period
10-11 NWEAFall Reading
Test RIT Score
10-11 NWEASpring Reading Test RIT Score
10-11 NWEATest RIT Score
Growth
10-11 NWEAFall Reading
Test Percentile
10-11 NWEASpring Reading Test Percentile
10-11 NWEATest Percentile
Growth
7 207 206 -1 15 18 3
2 207 219 12 10 40 30
2 212 202 -10 16 9 -7
1 207 219 12 15 50 35
2 210 213 3 14 25 11
2 220 216 -4 28 32 4
6 226 219 -7 53 50 -3
7 221 214 -7 41 35 -6
6 227 214 -13 56 35 -21
7 225 226 1 51 70 19
2 224 223 -1 36 52 16
6 231 202 -29 66 13 -53
6 232 213 -19 68 33 -35
56
College Readiness Benchmark Scores
Early Indicators of College Readiness
EXPLORE PLAN ACT
English English Composition 13 15 18
Reading Social Sciences 15 17 21
Math Algebra 17 19 22
Science Biology 20 21 24
ACT SubjectArea Test College Course(s)
Using Multiple Measures for Educational Decisions
Conjunctive Approach
(All measures count)
Measures of different constructs
College Readiness based upon student
achievement meeting identified benchmark
targets in English, Math, Reading, and
Science
Source: College and Career Readiness WORKSHOPS, Fall 2011
Rigor Issues
National HS Grad Class 2011 Profile Summary Report
CRB=22
Using PLAN to Predict ACT
• Uses 10th grade PLAN scores from 10-11• Use predicted ACT scores in each subject area• Use color-coding to indicate probability
– Dark Green– Light Green– Yellow– Orange– Red
Using EXPLORE to Predict PLAN
• Uses EXPLORE scores from 2010-2011• Use predicted PLAN scores
in each subject area• Use color-coding to indicate probability
– Dark Green– Light Green– Yellow– Orange– Red
Secondary Principals Study College and Career Readiness
10-11 PLAN Expected ACT 10-11 PLAN Expected ACT 10-11 PLAN Expected ACT 10-11 PLAN Expected ACT
Last name First name English English Reading Reading Mathematics Mathematics Science Science
16 16-20 17 17-21 14 14-18 16 16-20
17 17-21 17 17-21 20 21-25 21 22-26
22 23-27 20 21-25 21 22-26 21 22-26
16 16-20 14 14-18 16 16-20 20 21-25
10-11 Expected 10-11 Expected 10-11 Expected 10-11 Expected
EXPLORE PLAN EXPLORE PLAN EXPLORE PLAN EXPLORE PLAN
Last name First name English English Reading Reading Mathematics Mathematics Science Science
9 10-13 11 12-15 11 12-15 11 12-1514 15-18 12 13-16 14 15-18 15 16-1914 15-18 14 15-18 17 18-21 16 17-2013 14-17 12 13-16 15 16-19 16 17-2013 14-17 10 11-14 11 12-15 13 14-17
Reasonable Growth
• “On Target” • (met or exceeded CRB)
• “Nearly On Target” • (<2 points from CRB)
• “Off Target” • (>2 points from CRB)
Average Growth PointsBetween Tests
“On Target” (met or exceeded CRB)
“Nearly On Target” (<2 points from CRB)
“Off Target” (>2 points from CRB)
Test EXPLORE to PLAN
PLAN to ACT
EXPLORE to PLAN
PLAN to ACT
EXPLORE to PLAN
PLAN to ACT
English 2-3 2-3 3-4 1 3-4 1
Math 2-3 2-3 1-2 1-2 2-3 1-2
Reading 1-2 4-5 3-4 2-3 3-4 1-2
Science 1-2 2-3 1-2 2-3 2-3 1
Average Growth PointsBetween Tests
“On Target” (met or exceeded CRB)
“Nearly On Target” (<2 points from CRB)
“Off Target” (>2 points from CRB)
Test EXPLORE to PLAN
PLAN to ACT
EXPLORE to PLAN
PLAN to ACT
EXPLORE to PLAN
PLAN to ACT
English 2-3 2-3 3-4 1 3-4 1
Math 2-3 2-3 1-2 1-2 2-3 1-2
Reading 1-2 4-5 3-4 2-3 3-4 1-2
Science 1-2 2-3 1-2 2-3 2-3 1
Analysis of EXPLORE to PLAN
• English and Reading – Stronger relationship
• if low in English, then low in Reading
– language usage vs. reading comprehension?• Math achievement is low
– 8th grade math vs. Algebra I?• Low achievement in Science
– 8th grade Science course?
Analysis of PLAN to ACT
• Stronger relationship in Reading in English– curriculum vs. test?– lower CRB scores?
• Strongest relationship in Math– instruction in Algebra II?
• Weaker relationship in Science– process vs. content?
• Very few perform well on PLAN and miss CRB on ACT
Time Elements of an Assessment CalendarSource: White, S. H. (2005). “Beyond the Numbers: Making Data Work for Teachers and School
Leaders”. Lead and Learn Press: Englewood, CO
• When will we administer the assessment?• When will we collect the data?• When will we disaggregate the data?• When will we analyze the data?• When will we reflect upon the data?• When will we make recommendations?• When will we make the decisions about the
recommendations?• When will we provide written documentation about the
decisions?• When will we share the data with other stakeholders?
Performance Level
Scaled Score
Understanding Assessment Reports
Domain/ Standard Score
Benchmark/ GLCE Score
Written Curriculum Alignment
Analysis of Performance
Task
Analysis of Student Learning
V
A
L
I
D
I
T
Y
PLAN and EXPLORE Item Analysis
• Use test booklets from 11-12 testing– Order extra materials for
your staff (no cost)– Review items from the
booklet and the student responses
College Readiness Benchmark Standards
• Compare Standards in each subject area– below, at, and beyond benchmark
• Review written and taught curriculum– Which unit in the course/grade level?– How was it assessed in the classroom?– What were the students’ scores on the classroom
assessment?• Explore lesson plans and activities
Identify students who need assistance with the testing formats
• Needs identified by students on PLAN test
– Writing– Reading– Math– Study Skills
• Identify students who need assistance with the testing formats
– Writings using ACT rubric– Analyzing data in graphs,
charts, and tables– Use of released items from
MDE– Use of release practice items
from ACT– Strategies for completing
timed portions of ACT – Close and critical reading
strategies from MS/HS Literacy Team
College and Career Readiness WORKSHOPS
What are the academic achievements, educational
plans, and parental educational attainments
of our students?
Procedures for Reports
• Current student in 2011-2012:– Grade 10 or 11 for PLAN reports– Grade 9 or 10 for EXPLORE reports
• Valid PLAN or EXPLORE in 2010-2011• Selected self-reported data sets related to
college and career readiness
PLAN Student Profile Report
• Test Scale Scores – color coded• Self-Reported GPA on 4.0 scale• Self-Reported Post-Secondary plans• Self-Reported High School Coursework• Self-Reported Educational Attainment:
– Mother and Father
EXPLORE Student Profile Report
• Test Scale Scores – color coded• Self-Reported grades on coursework
aligned with test scores• Self-Reported Post-Secondary plans
Secondary Principals Study Student Profiles
Last FirstPLAN
EnglishPLAN
ReadingPLAN Math
PLAN Science GPA Plans HSCourses Ed Level - Mom Ed Level - Dad
14 22 17 17 1.6OTJT with military
College prep Some college >4-yr degree
21 24 23 23 3.4 4-yr coll./uni.College
prep Some college CTE
9 11 16 15 1.5 Undecided CTE/Other I don't know I don't know
10 13 15 16 2.8Nothing after
HS CTE/Other 2-yr degree HS/equivalent
19 17 18 21 1.9 >4-yr degreeCollege
prep Some college HS/equivalent
21 21 18 21 3.9 >4-yr degreeCollege
prep 2-yr degree HS/equivalent
20 18 18 21 2.3 CTE school CTE/Other 4-yr degree 2-yr degree
Questions to Consider
• What is the relationship between CCR test scores and grades?
• What is the relationship between CCR test scores and post-secondary plans?
• What is the relationship between post-secondary plans and parents’ educational attainment?
What are some other data sets to consider for “College and Career Readiness”?
• Standards Met• Credits Earned• Courses Taken• Grades Earned
• EDP Completion• Resume Completion• Interview Completion• Application Completion
• Open an EXPLORE, PLAN, or Elementary Report in Excel
• Determine College Readiness with color coding
Cut Scores, Student Growth and College/Career Readiness: 2011-2012 Data Dialogues
Stan MastersLenawee ISD
January 17, 2012
Goals For Today
• Describe how the new cut scores will impact the MEAP/MME– Refine Data from 10-11 MEAP
• Describe how a growth model operates – Create an Assessment with Calculation for Growth
• Describe how multiple measures help define “College and Career Readiness” – Open an EXPLORE, PLAN, or Elementary Report
in Excel and color code College Readiness