school and teacher effects: a team effort sharon walpole university of delaware michael c. mckenna...
TRANSCRIPT
School and Teacher Effects:
A Team Effort
Sharon WalpoleUniversity of Delaware
Michael C. McKennaUniversity of Virginia
So how’d we do last year?
Let’s look at some data from the UGA external
evaluator’s report
Letter Naming FluencyKindergarten - End of Year
67.5
19.213.3
74
1610
0
20
40
60
80
Low Risk Some Risk At Risk
2004-2005 2005-2006
Phoneme Segmentation FluencyKindergarten - End of Year
70.6
21.67.7
82
135
0
20
40
60
80
100
Low Risk Some Risk At Risk
2004-2005 2005-2006
Nonsense Word FluencyKindergarten - End of the Year
70.7
16.5 12.8
77
13 10
0
20
40
60
80
100
Low Risk Some Risk At Risk
2004-2005 2005-2006
But WHY?
Think about your most effective and your least effective kindergarten teacher. They both have the same materials. They have the same professional support system. They both have the same reading block.
What is it that actually differs between the two and might lead to differences in achievement?
Phoneme Segmentation FluencyFirst Grade - End of the Year
76.3
22.1
1.6
85
141
0
20
40
60
80
100
Low Risk Some Risk At Risk
2004-2005 2005-2006
Nonsense Word FluencyFirst Grade - End of the Year
64.9
27.7
7.5
73
23
4
0
20
40
60
80
Low Risk Some Risk At Risk
2004-2005 2005-2006
Oral Reading FluencyFirst Grade - End of the Year
64.7
22.712.6
70
2010
0
20
40
60
80
Low Risk Some Risk At Risk
2004-2005 2005-2006
But WHY?
Think about your most effective and your least effective first-grade teacher. They both have the same materials. They have the same professional support system. They both have the same reading block.
What is it that actually differs between the two and might lead to differences in
achievement?
Oral Reading FluencySecond Grade - End of the Year
50.8
22.326.9
56
21 24
0102030405060
Low Risk Some Risk At Risk
2004-2005 2005-2006
But WHY?
Think about your most effective and your least effective second-grade teacher. They both have the same materials. They have the same professional support system. They both have the same reading block.
What is it that actually differs between the two and might lead to differences in achievement?
Oral Reading FluencyThird Grade - End of the Year
44.5
33.9
21.9
51
31
18
0102030405060
Low Risk Some Risk At Risk
2004-2005 2005-2006
But WHY?
Think about your most effective and your least effective third-grade teacher. They both have the same materials. They have the same professional support system. They both have the same reading block.
What is it that actually differs between the two and might lead to differences in achievement?
Individual Differences
Teacher Differences
School
Differences
Portion of
variance in
achievement
explained
Lessons from School Effectiveness Literature
An LC can form judgments about teacher differences. It’s harder to judge differences at the school level because most of us lack the opportunities to make such comparisons.
Nevertheless, factors at the school level can have strong effects on achievement. The better we understand them, the more we can control them through leadership.
Differences among teachers have a clear impact on learning. Coaching attempts to reduce some of these differences by guiding teachers toward best practice.
But what about differences among schools? Can school factors be important as well?
What are some differences among kids that affect achievement?
What are some differences among teachers that affect achievement?
What are some differences among schools that affect achievement?
Organize Your Hunches
Child Teacher School
K
1
2
3
Let’s look at three studies that attempted to identify school effects on reading achievement. What can we can learn from them?
Beat-the-Odds Study*
Design FeaturesLooked at school and teacher factorsUsed measures of word readings, fluency, and retellings14 schools, 11 of which “beat the odds”2 teachers at each grade, and 4 children per classroomRelied on interviews, surveys, observations, and scores
School factors positively correlated with growthForging links with parentsUsing systematic assessmentFostering communication and collaboration
*Taylor, Pearson, Clark, & Walpole, 2000.
State-Level Outlier Study*
Design Features2 high-achieving, 1 low-achieving from 3 clusters: country, main street, uptownUsed state tests, interviews, and observations
School factors present in the high-achieving schoolsStrong leadership and commitmentTeacher knowledgeTime and opportunity for children to readCommitment of 8-10 years to the change process
*Mosenthal, Lipson, Torncello, Russ, & Mekkelsen, (2004).
Curriculum Effects Study*
Design FeaturesCompared 4 major reform models in first gradeEach used a coach, lots of PD, and regrouping of kids4 experienced schools in each model Measures of decoding, comprehension, vocabulary; classroom observations
ComparisonsAll of the schools were relatively successfulNone of the 4 models proved best
*Tivnan & Hemphill, 2005
What lessons do these studies teach?
Common Characteristics
1. No one curriculum or intervention model is a magical solution to student achievement problems
2. Intense focus on the school’s goals is associated with success
Assessment, communication, collaboration
Leadership, vision, knowledge PD plus specific curriculum
Differences From Us
1. These schools were successful or experienced already; we are striving to be successful and we are still new to our curricula
2. These schools had different demographics than we do (except for the Curriculum Effects Study)
3. These studies do not include the effects of intensive interventions
Limits of Generalizability
1. We can’t tell whether these characteristics are causes or characteristics of success
2. We don’t know whether these factors “transfer” to striving schools
Now let’s look at a fourth study – one with better lessons for Georgia and Reading First.
GA REA Study*
Question
In a two-year reform initiative, what school-level characteristics are significantly related to first-grade achievement?
Sample
22 striving schools in REA(Mean FRL = 73%)1146 first-grade children in those schools -- all children with full data
*Walpole, Kaplan, & Blamey (in preparation).
Data
Standardized measures of decoding and comprehension, DIBELS LNF and ORFInterviews, surveys
Findings
All of the schools were relatively successful; ORF mean, grade 1 = 60 wcpmStandardized scores above 50th percentile! Assessment-based planning explained a small but significant amount of the variance in first-grade achievement. This is a school factor!
So What IS Assessment-Based Planning Anyway?
Schools with higher achievement were rated higher on these two characteristics:
The leaders made thoughtful choices to purchase commercial curriculum materials to meet school-level needs.
The leaders of this project designed a comprehensive assessment system that teachers used to differentiate instruction.
How Did We Find Out?
We made a special rating sheet to rate the levels of implementation on all aspects of the project.
We correlated those ratings with student achievement to find the ones that were most powerful.
How Did We Find Out?
Four variables “survived” the correlations1. Differentiation strategies for word
recognition and fluency2. Strategy and vocabulary instruction during
read-alouds3. Careful choice of instructional materials4. School-level design of an assessment
system linked to instruction
In our RF work . . .
These variables correspond to:1. Differentiated word recognition
strategies for needs-based work2. Interactive read-alouds of children’s
literature (Beck + Duffy)3. Use (or purchase) of curriculum
materials based on their match with emergent achievement data
4. Selection of assessments that are really used to plan needs-based instruction
How did we find out?
These 4 variables were highly correlated with one another; we combined the two leadership variables and the two differentiation variables.
We controlled for LNF at January of kindergarten, we controlled for SES, and assessment-based planning was still correlated with first-grade scores at the school level.
What Do We Need to Learn?
We need to see what variables are powerful in GARF
We need to add the level of the teacher– Identify the characteristics we are targeting – Collect meaningful observational data on them
to see whether they make a difference We need to use the data we have and get
the data we need
That’s a tall order!
(But we think we cantackle it together.)
From a design standpoint
We are all trying to collect student data to measure the success of our programs.
It does not make sense to measure program effects without measuring treatment fidelity.
It does not make sense to measure treatment fidelity without observing the treatment.
It does not make sense to document treatment fidelity without trying to improve it.
Why?
Let’s look at the concept of “innovation configuration.” This is a way of finding out how fully we are implementing Reading First.
Innovation Configuration*
Full implementatio
n
Partial implementation
No Implementatio
n
The target practice is described here.
A practice in between (or, more likely, several different ones) is described here.
A description of a practice inconsistent with the target is described here.
*Hall & Hord, 2001
Context
http://www.nsdc.org/standards/index.cfm
Context Process Content
Learning communitiesLeadershipResources
Data-drivenEvaluationResearch-basedDesignLearningCollaboration
EquityQuality teachingFamily involvement
Moving NSDC's Staff Development Standards into Practice: Innovation Configurations
By Shirley Hord, Stephanie Hirsh & Patricia Roy
Innovation Configuration for Teacher’s Professional Learning
Level 4
Engages in collaborative interactions in learning teams and participates in a variety of activities that are aligned with expected improvement outcomes (e.g., collaborative lesson design, professional networks, analyzing student work, problem solving sessions, curriculum development).
Innovation Configuration for Teacher’s Professional Learning
Level 2
Attends workshops to gain information about new programs and receives classroom-based coaching to assist with implementation of new strategies and activities that may be aligned with expected improvement outcomes.
Innovation Configuration for Teacher’s Professional Learning
Level 1
Experiences a single model or inappropriate models of professional development that are not aligned with expected outcomes.
Procedure for Making an IC
1. Designer of an innovation describes ideal implementation of various components
2. Those “ideals” are compared with “real” implementation through observation
3. The “reals” are lined up from least like the ideal to most like the ideal
4. Then the IC can be used for observations, and even linked to student achievement!
Here are our classroom-level
“ideals” for GARF so far.
Physical Environment
The classroom is neat, clean, and organized so that the teacher can monitor all children and accomplish whole-group and needs-based instruction and so that children can get the materials they need. Wall space is used to display student work and curriculum-related materials that children need to accomplish tasks.
Curriculum Materials
There is one core reading program in active use. There is physical evidence of coherence in the text-level and word-level skills and strategies targeted in the classroom environment. Texts and manipulatives for whole-group, small-group, and independent practice are organized and available.
Children’s Literature
There is a large classroom collection of high-quality children’s literature deliberately organized and in active use that includes narratives, information texts, and multicultural texts.
Instructional Schedule
There is a posted schedule inside and outside the classroom to define an organized plan for using curriculum materials for whole-group and needs-based instruction; teacher and student activities correspond to the schedule
Assessment System
There is an efficient system for screening, diagnosing specific instructional needs, and progress-monitoring that is visible to the teacher and informs instructional groupings, the content of small-group instruction, and a flexible intervention system. All data are used to make instruction more effective.
Whole-Group Instruction
Whole-group instruction is used to introduce new concepts and to model strategies. Children have multiple opportunities to participate and respond during instruction.
Small-Group Instruction
Small-group instruction is used to reinforce, reteach, and review. Each child spends some time in small group each day; and small-group instruction is clearly differentiated. Children have multiple opportunities to participate and respond during instruction.
Independent Practice
Children work alone, in small groups, or in pairs to practice skills and strategies that have been previously introduced. They read and write during independent practice. They do this with a high level of success because the teacher organizes independent practice so that it is linked to whole-group and small-group instruction.
Management
The classroom is busy and active, but focused on reading. Classroom talk is positive and academic, including challenging vocabulary. Children know how to interact during whole-class, small-group, and independent work time. Very little time is spent teaching new procedures.
There’s a Lot We Don’t Know!
Is it possible to observe with one form across all grades?
How could we collect these observational data reliably and efficiently?
Which of these might explain variance in student achievement?
Procedure for Making an IC
1. Designer of an innovation describes ideal implementation of various components
2. Those “ideals” are compared with “real” implementation through observation
3. The “reals” are lined up from least like the ideal to most like the ideal
4. Then the IC can be used for observations, and even linked to student achievement!
Professional Support System*
*Joyce & Showers, 2002
Why?
Practice
Feedback Theory
Demonstration
You be the rater!
We will give you the IC that we made for the Georgia REA study.
Think about your first-grade team last year.
Provide us a realistic reflection on where you stood during Year 2.
We will use that data to analyze the Year 2 scores and provide you an update at our next meeting!
ReferencesHall, G. & Hord, S. (2001). Implementing change: Patterns, principles, and potholes.
Boston: Allyn and Bacon.Joyce, B., & Showers, B. (2002). Student achievement through staff development.
White Plains, NY: Longman. Mosenthal, J., Lipson, M., Torncello, S., Russ, B., & Mekkelsen, J. (2004). Contexts
and practices of six schools successful in obtaining reading achievement. The Elementary School Journal, 104, 343-367.
Taylor, B. M., Pearson, P. D., Clark, K. M., & Walpole, S. (2000). Effective schools and accomplished teachers: Lessons about primary-grade reading instruction in low-income schools. The Elementary School Journal, 101, 121-165.
Tivnan, T., & Hemphill, L. (2005). Comparing four literacy reform models in high-poverty schools: patterns of first-grade achievement. The Elementary School Journal, 105, 419-441.
Walpole, S., Kaplan, D., & Blamey, K. L. (in preparation). The effects of assessment-driven instruction on first-grade reading performance:Evidence from REA in Georgia.