using data to drive change mag conference november 2006
Post on 21-Dec-2015
214 views
TRANSCRIPT
![Page 1: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/1.jpg)
Using Data to Drive Change
MAG ConferenceNovember 2006
![Page 2: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/2.jpg)
How do we use data to influence decisions?
• School Level by Bruce Hislop, PGCPS
• Program Level by Clare Von Secker and Steve Bedford, MCPS
• Instructional Level by Carolyn Wood, HCPS
and Steve Perakis, CCPS
• Interventions by Bob Lissitz, MARCES, UMCP
![Page 3: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/3.jpg)
Determining Reasonable AMO Targets for Schools Far Above or Below State
AMO TargetsBruce Hislop
Accountability Reporting Officer, PGCPSMaryland Assessment Group Conference
November 16, 2006
![Page 4: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/4.jpg)
AMO Scenarios For A Given School
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
![Page 5: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/5.jpg)
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
SY07 Target For A “Yellow” School
![Page 6: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/6.jpg)
SY07 Target For A “Yellow” School
AMOSYT 0707
![Page 7: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/7.jpg)
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
SY07 Target For A “Yellow” School
![Page 8: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/8.jpg)
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
SY07 Target For A “Pink” School
![Page 9: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/9.jpg)
SY07 Target For A “Pink” School
AMOSYT 0707
![Page 10: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/10.jpg)
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
SY07 Target For A “Pink” School
![Page 11: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/11.jpg)
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
SY07 Target For A “Green” School
![Page 12: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/12.jpg)
SY07 Target For A “Green” School
CY
SYSYT
2014
061000607
5.8220062014
801008007
T
![Page 13: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/13.jpg)
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
SY07 Target For A “Green” School
![Page 14: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/14.jpg)
20
30
40
50
60
70
80
90
100
SY05 SY06 SY07 SY08 SY09 SY10 SY11 SY12 SY13 SY14
SY07 Target For A “Red” School
![Page 15: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/15.jpg)
SY07 Target For A “Red” School
Identification of “Red” Schools
– Distance below SY06 AMO
– Level in School Improvement (RP, RI)
![Page 16: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/16.jpg)
SY07 Target For A “Red” School
Three “Four-year-to-target” Models
1) Linear Growth
2) 70-70-85-100
3) 50-50-75-100
![Page 17: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/17.jpg)
Linear Growth
4
0610 PerfSYAMOSYthAnnualGrow
SY07- SY10 Targets For A “Red” School
![Page 18: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/18.jpg)
20
30
40
50
60
70
80
90
100
SY06 SY07 SY08 SY09 SY10
SY07- SY10 Targets For A “Red” School
Linear Growth
![Page 19: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/19.jpg)
70-70-85-100
70.04
061007 x
PerfSYAMOSYGrowthSY
70.03
071008 x
PerfSYAMOSYGrowthSY
85.02
081009 x
PerfSYAMOSYGrowthSY
PerfSYAMOSYGrowthSY 091010
SY07- SY10 Targets For A “Red” School
![Page 20: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/20.jpg)
20
30
40
50
60
70
80
90
100
SY06 SY07 SY08 SY09 SY10
SY07- SY10 Targets For A “Red” School
70-70-85-100
![Page 21: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/21.jpg)
50-50-75-100
50.04
061007 x
PerfSYAMOSYGrowthSY
50.03
071008 x
PerfSYAMOSYGrowthSY
75.02
081009 x
PerfSYAMOSYGrowthSY
PerfSYAMOSYGrowthSY 091010
SY07- SY10 Targets For A “Red” School
![Page 22: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/22.jpg)
20
30
40
50
60
70
80
90
100
SY06 SY07 SY08 SY09 SY10
SY07- SY10 Targets For A “Red” School
50-50-75-100
![Page 23: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/23.jpg)
Which Model Is Right For You?
Model is determined by levels of challenge and opportunity:
1) Percent FARMS
2) Percent classes taught by HQT
3) Years in principalship
![Page 24: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/24.jpg)
Which Model Is Right For You?
Degree of each factor is weighted, giving a weighted option scale.
% FARMS% Classes taught
by HQTYears in
Principalship
ValueWeigh
tValue
Weight
ValueWeigh
t
< 35% 0 > 74% 0 > 5 0
35.1 – 45.0
1 65 – 74 1 4 – 5 1
45.1 – 60.0
2 45 – 64 2 2 – 3 2
> 60.0 3 < 45 3 0 – 1 3
![Page 25: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/25.jpg)
Which Model Is Right For You?
Sum of weights determines accountability model
Sum 0 – 3 4 – 6 7 – 9
Option Linear70-70-85-
10050-50-75-
100
![Page 26: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/26.jpg)
This plan is under development and will most likely undergo more revisions.
![Page 27: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/27.jpg)
Using PSAT Data to Drive Instructional
Change
Clare Von Secker, Ph.D. and Stephen L. Bedford
Montgomery County Public SchoolsNovember 16, 2006
![Page 28: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/28.jpg)
Goals of PSAT Testing
• Preparation for the SAT• Identify students who
– Need additional support • Usually score range of 20 to 35
– Have Honors/AP potential• At or above mean verbal score of 44• At or above mean math score of 45
![Page 29: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/29.jpg)
PSAT Honors/AP Identification
• Identified as many as 2,000 sophomores per year who had Honors/AP potential but– Were enrolled in regular-level English
and mathematics courses in Grade 10– Were still enrolled in regular-level
English and mathematics courses in Grade 11
![Page 30: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/30.jpg)
Identify Other Considerations/Factors/Barrier
s• Enrollment decisions take into
account students’– Enrollment in other Honors courses– Academic performance– Attendance – Interests, motivation, and
recommendations
![Page 31: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/31.jpg)
Data Management Needs
• Decision-makers needed a way to – Compile all relevant student Honors/AP
indicators– Sort quickly and efficiently through
large amounts of student-level information
– Get buy-in from stakeholders (teachers, students, parents)
![Page 32: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/32.jpg)
Key Indicators• Stakeholders identified
– PSAT participation – PSAT verbal, math, and writing scores– Semester 1 enrollment in Honors-level core
courses – Total number of semester 1 Honors courses – Semester 1 English courses and marks– Semester 1 mathematics courses and marks– Grade point average (GPA)– Attendance rate – Participation in services for limited English
proficiency and special education– Gender and race/ethnicity (to assure equity)
![Page 33: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/33.jpg)
Student variables are listed as column headers in EXCEL
The column headers included for each school are:
• Grade • High school attended• Student last name, first name, and MCPS id • Yes/No “flag” showing whether student took the PSAT in Grade 10• PSAT verbal, math, and writing scores• Yes/No “flags” showing enrollment in Honors-level English, mathematics, science, social studies, and any course• Total number of Honors core courses taken semester 1• Semester 1 English and mathematics courses and final marks• Grade point average (GPA)• Attendance rate• Gender and race/ethnicity• Yes/No flag showing participation in services for limited English proficiency or special education• Name of student’s guidance counselor• Columns for counselors notes
![Page 34: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/34.jpg)
Filters identify all values of each variable
The values for the variable showing students’ Semester 1 Grade 10 math course include
• Below Algebra 1A (remedial mathematics courses)• Algebra 1A• Geometry A• Honors Geometry A• Algebra 2A• Algebra with Analysis 2a (Honors Algebra 2)• Precalculus• Precalculus with Analysis (Honors Precalculus)
![Page 35: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/35.jpg)
![Page 36: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/36.jpg)
How many of the students with PSAT scores of 45 or higher are not enrolled in Honors mathematics during the first semester (Honors math = No)?
The search found 97 out of a total 552 students in the sophomore class who should be considered for Honors-level mathematics in Grade 11.
![Page 37: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/37.jpg)
The search found 33 with PSAT math scores greater than or equal to 45 who were not enrolled in any Honors course during the first semester.
![Page 38: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/38.jpg)
The search found 18 students with • PSAT math scores greater than or equal to 45• not enrolled in any Honors course during the first semester• with GPAs greater than or equal to 2.5.
![Page 39: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/39.jpg)
Hispanic female student• PSAT scores of 64 (verbal), 54 (math), and 62 (writing)• not enrolled in any Honors course• GPAs greater than or equal to 2.5 • C in English 10• D in Algebra 2
![Page 40: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/40.jpg)
![Page 41: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/41.jpg)
Mastering Algebra: Using Teacher-Made Assessments to
Support Teaching and Learning
What We Can Learn About Students and Instruction From Locally-Developed Mid-Term
Exams
![Page 42: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/42.jpg)
Meeting the H S A Challenge
• Beginning with students entering grade 9 in September, 2005, all students must pass four High School Assessments, including Algebra/Data Analysis, or earn a minimum score on each and a Composite Score as a prerequisite to earning a Maryland high school diploma
![Page 43: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/43.jpg)
What We Know
• Students enrolled in grades 7, 8, and 9 during 2005-06 must PASS H S A BUT
• In 2006, only 12% of our 7th graders took and passed Algebra and only about 33% of our eighth graders had taken and passed Algebra
• In the past, about 40% of students have completed 9th grade without having passed Algebra
![Page 44: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/44.jpg)
The Scope of the Challenge in Algebra: Where We Stand
• AS OF SEPTEMBER, 2006—• Current 10th Graders
– 1800 students have passed H S A Algebra
– 1300 students have yet to pass H S A
• Current 9th Graders– 1200 students have passed H S A
Algebra– 2000 students have yet to pass H S A
![Page 45: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/45.jpg)
Math Skill among Middle School Algebra Takers is High:
2006 DataGrade Level 7 8
% MSA Math Advanced
63% 46%
% MSA Math Proficient
37% 45%
% MSA Math Basic
0 9%
![Page 46: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/46.jpg)
Math Skill Level of Studnts Taking Algebra in High School
is Low MSA MathGrade 8 Proficiency Level
Algebra Takers—Grade 9 (N=1048)
Algebra Takers—Grade 10 (N=692)
Advanced 50 (5%) <5
Proficient 530 (51%) 87 (13%)
Basic 468 (45%) 603 (87%)
![Page 47: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/47.jpg)
Success in Algebra: What It Takes
• Curriculum aligned to Core Learning Goals
• Instruction aligned with research-based best practice and the individual needs of students
• Frequent assessment that is FORMATIVE to inform instruction and motivate student effort
![Page 48: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/48.jpg)
Course Pathways to Mastery of Algebra 1
• Fully prepared middle school students complete Algebra in ONE YEAR (Grade 7)
• Somewhat prepared middle school students complete standard Algebra course in TWO YEARS (Grades 7-8 or Grades 8-9)
• Struggling students (including those who previously failed H S A) complete Cognitive Tutor Algebra course in Two Years (Grades 8-9 and Grades 9-10) – NOTE Cognitive Tutor was introduced for the
first time last year in some schools and grade levels
![Page 49: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/49.jpg)
Cognitive Tutor
• Based on cognitive model that simulates student thinking and problem-solving in math
• (Partially) Computer-delivered• Individualized to meet needs of student• Continuing assessment to support
diagnostic-prescriptive teaching• Simulates one-on-one coaching• Meets “scientifically-based research”
requirements of No Child Left Behind
![Page 50: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/50.jpg)
Multiple Roles of Assessment
• Certify student attainment of standard – Individual students– Programs (based on aggregated scores)
• Identify students for intervention– Placement in an “intervention program”– Instructional assistance as needed
• Inform teachers’ instructional decisions (pacing, delivery, amount and kind of content)– Group item-level data—item analysis, item
discriminations– Individual item-level data—items and item clusters
• Evaluate and improve instructional programs– Success rates on local end-of-course tests – Success rates on state tests
![Page 51: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/51.jpg)
Building Mid-Term Exams
• Mathematics Office needed assessments aligned with curriculum and H S A to– Provide practice to students– Show teachers what students need to
know– Intervention needs– Contribute to student course grade– Influence instruction
![Page 52: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/52.jpg)
Examination of Two Locally- Developed Assessments
• Algebra and Cognitive Tutor tests administered as mid-term exams
• Tests reflect different skill sequences • BOTH focus on Indicators within Goal
1(Functions and Algebra) and Goal 3 (Collect, Organize, Analyze, and Present Data)– Proportion of items aligned with Indicators varies
• Both include 20 SR items, 10 SPR items, and 3 or more CR items
• Both given under H S A-like conditions
![Page 53: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/53.jpg)
Reviewing the Products
• Are we measuring anything?• Are we measuring what we intend to
measure?• How do we know?
![Page 54: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/54.jpg)
Key Question
• How well do two mid-term exams that are designed for students enrolled in Algebra 1 and Cognitive Tutor Algebra, constructed by teachers, and aligned with state learning outcomes predict student success on H S A and provide useful feedback to guide teachers’ instructional decisions for individuals and groups?
![Page 55: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/55.jpg)
Sub-Questions• How reliable are the tests as instruments?• How valid are the tests for formative and summative use?
– FROM A CONTENT/CONSTRUCT Perspective• To what degree are the tests aligned with H S A/CLGs (content
validity)? With MSA?– Test construction
• Format and item construction– Test characteristics
• Item structure• Item-total relationships
– FROM A PREDICTION Perspective• Predicting success on H S A
– FROM A USER (INSTRUCTION) Perspective • Supporting instructional decisions• Informing students and parents
![Page 56: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/56.jpg)
Study Sample
• Algebra 1B
• 988 9th graders • 61% proficient on
MSA in Grade 8 (2005)
• 88% passed H S A in May, 2006
• Cognitive Tutor Algebra
• 45 9th graders • 16% proficient on
MSA in Grade 8 (2005)
• 34% passed H S A in May, 2006
![Page 57: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/57.jpg)
Are these instruments measuring anything?
• How reliable are they? – Evidence of Internal Consistency
• Cronbach alpha – Algebra 1B version: alpha= 0.82– Cognitive Tutor version: Alpha = 0.76
• Item-Total Correlations: 0.15 - 0.62
• What can we say?– Reliability modest but OK for a
classroom test
![Page 58: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/58.jpg)
What will the tests tell us?
• Inferences about student competence in algebra (overall) and the likelihood they will pass the H S A
• Inferences about students’ knowledge about the indicators
• Inferences about students’ ability to manage different kinds of items
![Page 59: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/59.jpg)
Are the Midterms measuring “Mathematics Knowledge?”
“Algebra/Data Analysis?”• Evidence from MSA and H S A Algebra• Algebra 1B Midterm (Grade 9 ONLY;
N=988)– Correlation with MSA Total (GR 8):0.58– Correlation with MSA ALGEBRA Subscore (GR
8): 0.46– Correlation with MSA Total (GR 7): 0.50– Correlation with MSA ALGEBRA (GR 7)
Subscore: 0.38– Correlation with H S A Algebra Total: r = 0.58– Correlation with H S A Algebra/SS1 Patterns
and Functions: r = 0.44
![Page 60: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/60.jpg)
Are the Midterms measuring “Mathematics Knowledge?”
“Algebra/Data Analysis?”• Evidence from MSA and H S A• Algebra/Cognitive Tutor Midterm
(Grades 9 and 10 9 ONLY; N=574)– Correlation with MSA Total (GR 8):0.45– Correlation with MSA ALGEBRA
Subscore (GR 8): 0.33– Correlation with H S A Algebra Total: r
= 0.50– Correlation with H S A Algebra SS 1
(Patterns and Functions) r = 0.46
![Page 61: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/61.jpg)
Cognitive Tutor Exam Parallels the H S A Topics and Item
Design– Items by Topic
• (1.1) Analyze patterns and functions—9 items• (1.2) Use language of math to model and
interpret real-world situations—19 items• (3.1) –Collect, represent, organize data--5
items
– Items by Type• 10 student-produced response• 20 selected response• 2 brief constructed response• 1 extended constructed response
![Page 62: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/62.jpg)
Algebra 1B Exam Parallels H S A Topics and Item Design
– Items by Topic• (1.1) Analyze patterns and functions—17 items• (1.2) Use language of math to model and
interpret real-world situations—10 items• (3.1) –Collect, organize, and present data--5
items• (3.2)– Apply statistics and probability in real
world—1 item
– Items by Type• 10 student-produced response• 20 selected response• 4 brief constructed response
![Page 63: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/63.jpg)
Evidence Supporting Interpretation of Item Clusters• Looked for correlations among items
purportedly measuring the same indicator: no consistent patterns
• Found some tendency for like items (formats) to cluster
![Page 64: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/64.jpg)
Evidence of Predictive Validity
• Algebra 1 B– Use Raw Score to Predict H S A ALG status– Adjusted R-squared: 0.341– Constant = 391.60; B = 1.31– SE Estimate: 19.51
• Cognitive Tutor– Use Raw Score to Predict H S A Algebra status– Adjusted R-square: 0.616– Constant = 353.05; B=2.27– SE Estimate: 18.08
![Page 65: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/65.jpg)
Success Rates for Ninth Graders
• Students enrolled in Algebra 1B– OVERALL PASS RATE: 88%– Per Cent of Students at MSA (GR 8)
BASIC Who Passed: 72%
• Students enrolled in Cognitive Tutor– OVERALL PASS RATE: 40%– Per Cent of Students at MSA BASIC (GR
8) Who Passed: 35%
![Page 66: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/66.jpg)
What We Learned • A reasonably reliable measure resembling H S A
and aligned with Content Standards can be constructed by teachers– SOME evidence that total scores are related to later HSS
performance and could help to predict student success– Teachers and students are exposed to a “test-like” event in
preparation for a high stakes test• More work is needed to construct an assessment
that provides useful formative feedback to teachers – No real evidence that items measuring Indicator groups
were related in predicted ways: Implications = ?– Some evidence of relationships between item types:
Implications = ?– What would useful “formative feedback” look like and what
kind of instrument/set of instruments would be necessary?
![Page 67: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/67.jpg)
From Prediction to Improvement: A Vision for
Classroom Assessment• The framework for assessment and the
framework for curriculum and instruction must be one and the same. . . Using progress variables to structure both instruction and assessment is one way to ensure that the two are in alignment at least at the planning level
• At the classroom level, [assessment tasks] must have a place in the rhythm of instruction, occurring at places where it makes instructional sense to include them. This is usually at points where teachers need to see how much progress students have made . . .
• Mark Wilson (2005)
![Page 68: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/68.jpg)
Next Steps• Improve quality and quantity of FORMATIVE data
– Consider best ways to provide formative data given the nature of the course, the students, the discipline (when, where, how, etc)
– Examine Indicators to be tested within context of instruction– Design/try-out/improve test items based on student
responses– Make available multiple assessment opportunities for
students– Assist teachers in scoring/evaluating student performance
• Review rationale for mid-term exams (e. g. explaining the demands of H S A to students and teachers, identifying students at risk on H S A, etc.), decide upon critical features of the exams, work to improve current forms e. G, – Structuring items that represent H S A content – Item formats represent H S A items
![Page 69: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/69.jpg)
Predicting HSA Performance with Public Release HSA
ItemsCharles County Public
SchoolsSteven Perakis
![Page 70: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/70.jpg)
Predicting HSA performance
• Charles County prepared Mock HSA Assessments, some 2 months prior to the HSA examinations.
• The tests mirrored the HSAs in length and we used test blueprints provided by MSDE.
• Student performance on the HSAs were tracked, and were merged with student performance on their examinations.
![Page 71: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/71.jpg)
• IRT calibration was undertaken on tests written by approximately 2000 students. The tests were comprised almost exclusively from public release items.
• The KR-21 on the examinations were approximately 0.92 and the correlations between IRT derived student theta scores and the MAY 2006 HSA examination scores were 0.80.
![Page 72: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/72.jpg)
• We attempted to predict HSA scores using multiple regression, logistic regression, and simple cross tabulations from “Mock examination” scores.
![Page 73: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/73.jpg)
Results:
• Multiple regression: Successfully predicted 53% of the students’ scores within +/- 10 scaled points and 81% of student HSA scores within +/- 20 scaled score points.
• Logistic regression: Using the student’s actual Pass_Flag on the HSA as the dependent variable, probabilities for each student passing each HSA were produced. Logistic regression produced probability tables for passing for each student. On the LSN Mock examination for example, above a probability for passing of .60, 93.5% of the students actually passed the spring HSA representing 78% of the students writing that examination.
![Page 74: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/74.jpg)
Thoughts and Conclusions
• Individualized student performance profile reports by Core Learning Goal, focused teachers in their test-preparation run-up to the HSAs.
• Charles County found that student performance on these tests comprised of public release items to be effective predictors of student performance on the 2006 HSAs.
![Page 75: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/75.jpg)
Future Study: HSA Interventions
Dr. Robert LissitzMARCES
University of Maryland College Park
![Page 76: Using Data to Drive Change MAG Conference November 2006](https://reader038.vdocuments.mx/reader038/viewer/2022110207/56649d635503460f94a45fae/html5/thumbnails/76.jpg)
HSA Studies
• Voluntary Collaborative Research Group between MARCES and School Systems (funded by MSDE)
• Phase 1: Predicting Performance on HSA
• Phase 2: Examining Interventions– What interventions exist that should be studied?– What studies are already underway?– What studies should be done?