improving teacher quality grants, cycle 4: external evaluation report december 7 th, 2007 university...
TRANSCRIPT
Improving Teacher Quality Grants, Cycle 4:
External Evaluation ReportDecember 7th, 2007
University of Missouri-Columbia Evaluation Team
Evaluation Team
Principal Investigators
Sandra Abell
Fran Arbaugh
Mark Ehlert
John Lannin
Rose Marra
Graduate Research Assistants
Kristen Hutchins
Ya-Wen Cheng
Zeynep Kaymaz
Michele Lee
Dominike Merle
Chia-Yu Wang
Context of the Evaluation
• Improving Teacher Quality Grant program, Cycle 4, 2006-2007– Focus on high-need schools– 14 funded professional development projects– Science and mathematics, grades 4-8
Participant Summary• 383 participants;
• 84.0% white and 84.2% female;
• Most held regular certification in elementary/early childhood (n=236) and/or middle level mathematics (n=122) and/or science (n=203);
• 29% had been teaching for 1-5 years, 22.7% for 6-10 years, 34.5% for 11-15 years, and 9.5% for 26 or more years;
• 64.7% had participated in some form of PD the previous year; however, few participated in mathematics or science-specific, district-sponsored PD;
• 62.6% were new to the ITQG program;
• Taught in 124 different Missouri school districts, 3 private schools, and 3 charter schools;
• Directly impacted 33,580 students in the 2006-2007 school year.
Percentage of Participants from High-Need Districts
44.2%
55.8%
% participants from high-need districts % participants from non high-need districts
Were Cycle 4 ITQG Objectives Met?Objective Data Responding to Each
•Improve student achievement in math/science
•Students learned new content (MAP scores, student content tests)
•Increase teachers’ content knowledge and improve teachers’ pedagogical knowledge and skills
•Teachers learned new content and instructional strategies, and some began to use more inquiry-based instructional practices (evaluation instruments, teacher content knowledge tests)
•Impact on pre-service teachers
•Higher ed institutions improved their preparation of future teachers (evaluation instruments)
•Enhance use of assessment •Teacher knowledge/use of assessment was not improved (evaluation instruments, teacher content knowledge tests)
Four Effective PD Practices
• PD content
• PD delivery
• PD sustainability
• PD as part of a system
PD Content--Effective PracticesWhat the research says:Effective PD provides opportunities for teachers to build their content and pedagogical content knowledge and examine practice. (Loucks-Horsley et al., 2003)
What the standards say:Professional development for teachers of science requires integrating knowledge of science, learning, pedagogy, and students; it also requires applying that knowledge to science teaching. (NRC, 1996)
What the Cycle 4 RFP said:•Provide content knowledge related to standards and classroom instruction.•Provide instructional strategies related to content being taught in the classroom.•Provide research-based instructional strategies to the participating teachers.
PD Content—What We Found1. Science and mathematics subject matter content
--Relevant to teachers--Less is more
2. Integration of science and mathematics content--Need for explicit connections
3. Integration of subject matter and pedagogy--PIs must make their expectations for teacher learning clear,
make their pedagogical models explicit, and integrate theory/practice as well as content/pedagogy.
PD Content--Participant Preferences
Teachers preferred when PD content:•Related to their grade level and subject
•Aligned with local curriculum and GLEs
•Reflected teacher knowledge needs
•Related to student knowledge needs
•Included both science/mathematics content and pedagogical content
Representative comments:“I liked the different project ideas that relate to the GLEs.”“Working with professors that could help us combine the GLEs with our lesson design was probably the most valuable part of the experience for me.”“The most valuable part was working in groups to create inquiry-based lessons that are aligned to the GLEs.”
PD Content—Relevant Learning
0=No gain in content knowledge1=None of the gain was relevant to my teaching assignment2=Very little was relevant3=A Moderate amount was relevant4=Most of the gain was relevant5=All of the gain was relevant
FrequenciesDescriptive Statistics
Subject area 0 1 2 3 4 5 n/a n M SD
Mathematics 7 1 19 47 59 39 4 172 3.6 1.2
Science 3 3 7 46 75 84 6 218 4.0 1.0
End of Summer Institute
PD Content—Science/Math Subject Matter Amount
• Projects delivered PD to teachers across a range of grade levels. – largest range: 6 grade levels (grades 3-8) – smallest range: 2 grade levels (grades 6-7)
• Projects listed 2-8 subject matter foci
• Teachers were more satisfied when fewer content areas were covered
How to Design for Effectiveness in Science/Math Subject Matter
• UCM—Grade-level appropriateness
• Lincoln—Less is more
PD Content—Science/Math Integration
• 14 Cycle 4 projects– 5 targeted science – 2 targeted mathematics– 7 seven targeted both science and
mathematics
PD Content—Science/Math Integration
What participants saw as “least valuable”• The higher level math since I am lower level
science. • I am an elementary teacher and many of the
lessons were designed for high school math. I did not find this helpful to me.
• This experience wasn't related to teaching math. I am a math teacher and this experience was hard to incorporate into my classroom.
• I am a math teacher exclusively, so the time spent on science was not well spent.
How to Design for Effective Science/Math Integration
• UMR—Science and mathematics integration
PD Content—Subject Matter and Pedagogy
Contribution of Professional Development Emphasis Areas to Practice Reported by Participants at End of Professional Development Projects
UEmphasis AreasU UnU UMU USDU
Pedagogy 220 7.6 1.9 Content knowledge 221 7.1 2.2
0=none to 10=very much
How to Achieve Effective Balance of Subject Matter and Pedagogy
• SEMO—Integrating pedagogy into math content week
• NMSU—The learning cycle and mathematics
• Rockhurst—The learning cycle and science
PD Delivery
What the research says:Effective PD is driven by a well-defined image of effective classroom learning and teaching (Loucks-Horsley et al., 2003)
PD Delivery
How to use a framework to guide effective PD:
• Lincoln
• Rockhurst
PD Delivery
Challenges:
• Diversity of project teams
• Shared vision develops over time
• Involvement of local school district personnel
PD Delivery What the research says:•Effective PD engages teachers as adult learners in the learning approaches they will use with their students. (Loucks-Horsley et al., 2003)Create a high level of cognitive dissonance, and Provide sufficient time, structure, and support for teachers to think through the dissonance experienced (Thompson and Zeuli, 1999)
What the Cycle 4 RFP said:Provide research-based instructional strategies to
participating teachers
PD Delivery
Table VI.4: About How Often Do STUDENTS in Your Class Take Part in Each of the Following Types of Activities as Part of their Mathematics/Science Instruction?
Frequencies Descriptive Statistics
0 1 2 3 4 n M SD
Participate in student-led discussions 1 6 11 17 6 41 2.5 1.0 Participate in discussions with the teacher to further math/science understanding 0 3 5 24 10 42 3.0 0.8
PIs and Project Staff, Cycle 4 (p. 38)
PD Delivery
Table VI.5: How Often Do You Assess Student Progress in Mathematics/Science in Each of the Following Ways?
Frequencies Descriptive Statistics
0 1 2 3 4 n M SD Conduct a pre-assessment to determine what students already know 0 10 14 14 4 42 2.3 0.9
PIs and Project Staff, Cycle 4 (p. 38)
0=Never; 1=Rarely; 2= Sometimes; 3=Often; 4=All/almost all lessons.
PD Delivery
Table VI.2: About How Often Do YOU Do Each of the Following in Your Mathematics/Science Instruction?
Frequencies
Descriptive Statistics
0 1 2 3 4 n M SD
Introduce content through formal presentations 2 3 17 15 5 42 2.4 1.0 Engage the whole class in discussions 0 0 10 16 16 42 3.1 0.8 Pose open-ended questions 0 1 3 23 15 42 3.2 0.7 Require students to explain their reasoning when giving an answer 0 1 8 12 21 42 3.3 0.9 Ask students to explain concepts to one another 0 5 2 22 13 42 3.0 0.9 Encourage students to explore alternative methods for solutions 0 1 9 15 17 42 3.1 0.8
PIs and Project Staff, Cycle 4 (p. 35)
0=Never; 1=Rarely; 2= Sometimes; 3=Often; 4=All/almost all lessons.
PD DeliveryTeacher comments (Cycle 4)
I learn really well with this hands-on approach, and until these workshops, I have had limited knowledge of physics, especially how to formally teach energy transfers, etc.
We were able to work through the activity in the role of the student and then received the content knowledge that went along with it. By doing the activity myself I was able to completely understand how to recreate the activity for my students during the regular classroom.
Learning in the mode we expect our students to learn in was a very valuable tool in my teaching. It allowed me to feel some of the frustration my students would later feel when I changed from being the "giver" of information to the facilitator of students "finding" the information during the inquiry part of each lesson. (p. 48)
PD Delivery
How to engage teachers in approaches they will use with their students:
• UMSL - literacy strategies
• MU - notebooking
PD Delivery What the research says:Effective PD provides opportunities for teachers to collaborate with colleagues and other experts to improve their practice. (Loucks-Horsley et al., 2003)
What the Cycle 4 RFP said:–Contain grade-level and/or content-area collaboration
and work–Provide teachers with the opportunity to reflect on their
practices and to give the district feedback on the effectiveness of participation in this activity/experience
PD DeliveryTeacher comments (Cycle 4)
•To me, as a fairly new teacher the chance to collaborate with other professionals for that amount of time on our unit was very valuable. I loved having the chance to get to know others and build a team situation rather than feeling like I was on my own.
• I think talking and visiting with other teachers was invaluable! Being able to collaborate with other classroom teachers that also taught my grade level! I am the only 4th grade teacher within my district and I enjoyed meeting and planning some lessons with other teachers who also taught 4th grade.
•Time to collaborate with other teachers, share teacher techniques, classroom management strategies, plan and share ideas for lessons -- all of these things allowed me to expand my practices and methods in the classroom
•Networking with other teachers on the projects. I think I gained more knowledge from them as we worked through the activities. It was great to talk to teachers and realize you had similar problems in your classes. (p. 46)
PD Delivery
A vehicle for teacher collaboration:
• MSSU--E-conferencing
PD Sustainability
What the research says:Effective PD builds capacity for sustainability
(Loucks-Horsley et al., 2003)
What the Cycle 4 RFP said:Proposal narrative should:…• discuss how teachers and other participants will
be actively engaged over the life of the project and the potential for the project’s sustainability beyond the end date of the grant
PD Sustainability
Sustainability in ITQG projects:
• Year-long sustainability
• Multi-year sustainability
• Post-project sustainability
Year-long Sustainability
What did you do in your Cycle 4 project to promote teacher engagement throughout the year?
Teachers’ Views
PD Delivery Mechanisms n M SD max/min
Summer institute 219 8.3 2.0 1/10
School-year call backs 221 7.1 2.5 0/10
Other school-based activities 213 6.6 3.1 0/10
0=no contribution to practice, 10=very much contribution.
Year-long Sustainability: Call-back Sessions
• The Saturday call-back sessions. Much of the time we were not engaged in a meaningful task.
• I came to the first follow-up session and I felt that I wasn't getting anything out of it. I guess this is why I chose not to come to most of the sessions.
• The call back sessions because of the stress created trying to juggle my extra-curricular school responsibilities to allow me time to come.
Year-long Sustainability:Classroom-based Support
• Focus on PD content
• Clarity about the purpose(s) of classroom-based support
• How to sustain classroom-based support:– UCM– UMKC1
PD Sustainability
• Multi-year sustainability– PD improvement over time– Repeat participants
• Post-project sustainability– Small number of districts to concentrate
impact– Multiple teachers from the same school
PD Sustainability
Challenges:• Participant attrition (within a year; from year to
year)• “Mile-wide, inch deep” approach to participant
recruitment (issues of scale)• District priorities• Continued funding
PD as Part of a System What the research says:• School culture, leadership and support, and time for teacher learning
influence the quality of PD (Garet et al., 1999; Sparks, 2002)• Effective PD links with other parts of the education system (Loucks-
Horsley et al., 2003)
What the Cycle 4 RFP said:• Directly link to district and building school improvement plans;• Be developed with extensive participation of teachers, principals, and
other administrators;• Provide time and other resources for learning, practice, and follow-
up;• Be supported by district and building leadership.
Sample Teacher Comments“Although I enjoyed meeting and discussing content and objectives with other teachers I found that planning and writing lessons … with teachers outside of my own district was not helpful. Each district has their own focus so it was hard to work it out.”
“Some of the lessons really did not apply to my district’s alignment.”
“This project was wonderful. I only regret that I was not able to use it to the fullest extent based upon decisions made by the district I worked for. “
PD as Part of a System
How to achieve various partnerships:• UMKC2—Partnership with KC schools at
planning phase• UMSL—Partnership of science and literacy• 3 Rivers—Key role of school-based personnel in
planning and instructional team (Also links to Math Academy)
• MSSU—ITQG project as part of larger PD effort• MSU—networking with rural schools
PD as Part of a System
Challenges for future cycles:• Improving participation of high-need districts• Forming partnerships with K-12
– During design phase– Around assessment issues– For sustainability
• Linking higher education components
Four Effective PD Practices
• PD content
• PD delivery
• PD sustainability
• PD as part of a system
Future of Missouri METS
Importance of connecting ITQG with other METS initiatives:– DESE-funded MSP projects– METS Alliance– Teacher Leadership Academy
For example:– CMU—Competition for middle school students
Questions
Copies of the report and Executive Summary available at:
www.pdeval.missouri.edu