Lessons Learned from Evaluating Science Education Projects

Dave WeaverRMC Research Corporation

111 SW Columbia, Suite 1200Portland, Oregon 97201dweaver@rmccorp.com

(800) 788–1887

2

Presentation Contents

• Lessons Learned from LASER Sentinel Site Visits

• 3 Common Pitfalls to Improvement– How LASER and Washington Science

Leadership is Responding

• Implications for Your Strategic Planning

3

Analysis of Science WASL Data

• Evaluation Question– To what extent did teacher professional

development on inquiry-based science instruction contribute to improved student achievement on the Grade 5 and 8 Washington Assessment of Student Learning of science (science WASL)?

4

Finding 1

• The number of professional development hours in which a student’s science teacher participated was a small but significant predictor of student performance on the science WASL above and beyond what could be explained by socioeconomics (FRL) and the student’s skill level (previous math WASL).

• This finding is consistent with earlier studies.

5

PD Has Modest Impact on Achievement

40.6% 41.3%

43.4%

30%

32%34%36%38%40%

42%44%

46%48%

50%

Adjusted Means

3-Year PD Per FTE

Per

cent

of

Gra

de 5

Stu

dent

s W

ho

Met

/Exc

eede

d Sc

ienc

e St

anda

rd

Less Than or Equal to6.4 Hours

Greater Than 6.4 andLess Than or Equal to15.75 hours

Greater Than 15.75hours

Means adjusted by previous year math scores, and FRL.

Lessons Learned from LASER Sentinel Site Visits

Results from the 2007-2009 LASER Biennium

7

Sentinel Site Selection• Schools with significant LASER participation

during the 3 years prior to the site visit• Schools visited

– 34 schools during 2007-08 school year– 30 schools during 2008-09 school year

• Used standard protocol and rubrics• Defined 2 groups of schools based on science

WASL change for 2 years prior to site visit – Demonstrated Significant Positive Gains– Demonstrated Little, No, or Negative Gains

8

Sentinel Site Visits• RMC Research staff or consultants• Each site visit: 1 ½ to 2 days

– Principal interview– At least 5 teacher survey & interviews– At least 3 classroom observations

• Data collection instruments– http://www.rmccorp.com/LASERSiteVisits/

• Conducted web-based training session• Results reported as rubric scores

9

Finding 2

• There were a number of traits that site visitors consistently rated high, many of which were a direct outcome of Washington State LASER.

10

What Was Going Well At The School Level• Materials Support System

– The system for maintaining, storing, and refurbishing the instructional modules was effective and well organized.

• Condition of Modules– Teachers always received modules that were complete and ready for classroom use.

• Inquiry-Based Materials– The school was implementing 3 or more modules per grade level as the core science

curriculum materials. Ninety-four percent of the teachers used the modules as the core science curriculum.

• Administrative Support– Strong evidence indicated that the school administrators were very supportive of inquiry-

based science instruction.• District Support

– Strong evidence indicated that the district administrators were very supportive of inquiry-based science instruction.

• Sequence– All science teachers used the modules according to a sequence clearly prescribed by

the district.• Critical Mass

– Most (80% or more) teachers in the school had attended the initial use training for all of the modules they used.

11

School Level Areas Receiving Low Scores• Summative District Assessments

– Very few school had districtwide or schoolwide summative assessments in science that were administered to students annually

• Formative Assessments– A few teachers (25% or less) had adopted a standard formative assessment

strategy for science.

• Instructional Time– Science instructional time varied considerably amount teachers at the

elementary level and few schools required a minimum amount of instructional time for science.

• Professional Development Time for Teachers– Teachers rarely had scheduled time during normal work hours to participate

in school-based professional development in science.

• Partnership With Business, Informal Science, or Higher Education– A few teachers had a tentative partnership with a business, an informal

science organization, or an IHE around science education.

12

Classroom Observation: High Scoring• Alignment of Lesson Activities

– Lesson activities addressed the stated learning objectives but there was some question about how the lesson activities would lead to a deeper student understanding of the learning objectives.

• Motivation– The lesson provided mostly extrinsic and some intrinsic motivation. The

intrinsic motivation was truncated by the lesson structure and was relatively short lived.

• Understanding of Purpose– Throughout the lesson, many students understood why they were doing

each activity but the purpose of activities could have been more explicit.• Classroom Discourse

– For the most part, students and teachers support and encourage respectful and constructive discourse around important science concepts, however, only some students feel comfortable asking questions, backing up their own claims, and/or critiquing claims made by others.

13

Classroom Observation: Low Scoring• Lesson Closure

– By the end of the lesson, the teacher provided a brief review, but students did not have an opportunity to fully make sense out of how the lesson related to science concepts.

• Application of Science– A few students applied something they learned in the lesson

to a new context.

• Reflection and Meta-cognition– By the end of the lesson, students had some opportunity to

reflect on their thinking but students were not asked to identify ways in which their thinking about the science concepts had changed.

14

Analysis of Gains• Divided the schools into 2 groups

– Based on:• Percent of students who met the science standard• Gains calculated over 2 years prior to the site visit

– Group definition• Schools that demonstrated an increase in student achievement• Schools that had no change or decreasing student achievement

• The 2 groups of schools had significant demographic differences

• Percent of students who qualified for free or reduced price lunch

• Percent of Asian students

15

Finding 3• Although schools that demonstrated

increasing student science achievement were significantly different demographically from those that did not, there were also significant differences in the instructional practices of the teachers observed by the site visitors.

16

Differences Between Gain Groups• Science classes in schools that demonstrated an

increase in student achievement were more likely to:– Have clear lesson objectives– Involve activities that clearly address the learning objective– Have students who understand the purpose of the lesson– Have students that are more intellectually engaged in the

science content– Have students applying science content to new contexts– Have students engaged in science discourse– Motivate students intrinsically

17

Analysis by Achievement Ranking• Divided the schools into 2 groups

– Based on percent of students who met the science standard the year of the site visit

– Group definition• Schools at or above the state average• Schools below the state average

• Methods– Regression analysis to determine which variables were

significant predictors of the achievement ranking– Controlling for

• Student achievement the previous year• Percent of students who qualify for free or reduced price lunch

18

Finding 4• Although schools that demonstrated

above average student science achievement were significantly different demographically from those that were below average, there were also significant differences in the characteristics of the schools.

19

Characteristics of Above Average Performing Schools

• Instructional Time Allocated– The elementary school has a designated amount of instructional time

allocated for science.• Professional Development Time for Teachers

– Teachers occasionally had scheduled time during normal work hours to participate in school-based professional development in science.

• District Support– Evidence indicated that the district administrators were somewhat

supportive of inquiry-based science instruction.• Integration of Literacy

– Many teachers integrated science and literacy through the use of supplementary reading materials and science notebooks.

• Parent and Community Support– Evidence indicated that the parents/community were somewhat supportive

of inquiry-based science instruction.

20

Principal Survey• Online survey that closely paralleled data collected

during sentinel site visits• Selected schools with 15 hours or more of science

professional development per teacher over a 5-year period prior to March 31, 2009

• 319 schools invited• Survey administered between May 22 and July 31,

2009• 62 principals completed the survey (19.4% return rate)

21

Principal Survey Analysis• Regression analysis to determine which

survey items were significant predictors of student achievement on the 2009 science WASL

• Controlling for the percent of students who qualified for free or reduced price lunch (FRL)

22

Finding 5

• There were several items on the principal survey that were significant predictors of student performance on the science WASL above and beyond what could be attributed to FRL.

23

Predictors of Student Performance• Schools that made an organized effort to identify

instructional materials to fill the gaps• Schools where the principals observe student using

evidence to engage in discourse about science.• Schools that provide time during the normal work day

for professional development and how often teachers participate in school-based science professional development.

• Schools that support professional learning communities that focus on improving science teaching and learning.

24

Regression Analysis ResultsSurvey Item

Grade Tested

Adjusted R2

Change in R2 p Beta

Has your school or district made an organized effort to identify instructional materials to fill the gaps?

5 .459 .114 <.001 * .352

Principal observation of classes: Students had opportunities to make claims, and/or use evidence to back up their claims or critique claims made by others. The lesson reinforced the notion that science is a process by which knowledge is constructed.

5 .563 .122 <.001 * .350

Is time scheduled during normal work hours for teachers in this school to participate in organized, school-based professional development specifically for science?

5 .338 .002 .003 * .048

How often do teachers participate in school-based professional development specifically for science?

5 .213 .017 <.001 * .153

Has any of the PLC activities focused on improving science teaching and learning?

5 .476 .031 <.001 * .211

Approximately what percentage of the PLC is devoted to science teaching and learning?

5 .434 .002 .013 * .047

25

Finding 6• There are several consistent themes

among the various findings that provide the basis for recommendations and conclusions that lead to a shift in Washington State LASER.

26

Significant School-Level Factors

• PD time for teachers during the school day• Allocated science instructional time• Professional learning communities• Filling curriculum gaps• Integration of literacy• District, parent, and community support

27

Significant Instructional Factors

• Science discourse using evidence

• Purposeful instruction

• Intellectual engagement

• Intrinsic motivation

• Application of science skills

28

Conclusion• The infrastructure to support the use of a core

curriculum of inquiry-based science instructional modules is in place and is functioning adequately in the schools visited.

• Although these conditions are necessary for the implementation of inquiry-based science instruction, they are not sufficient to raise student achievement as measured by the science WASL.

29

Recommendation 1• Ensure that the professional development on

research-based instructional practices is consistent and explicit across all of the LASER Alliances– Help teachers understand the elements of effective

science instruction and use the modules as a means of carrying out the element with their students.

30

Recommendation 2• Increase support for school-based

professional development that helps teachers:– Assume accountability for student learning

that results from the use of the modules, and

– Collaboratively implement the elements of effective science instruction.

– Ample structure and leadership for success

3 Common Pitfalls to Improvement

32

Common Pitfalls

• Why haven’t past school reform efforts been more successful?

• Three Common Pitfalls:– Lack of Vision

– Lack of Process

– Atomized Culture

33

Common Pitfall #1• Lack of Vision

– “In most instances, principals, lead teachers, and system-level administrators are trying to improve the performance of their schools without knowing what the actual practice would have to look like to get the results they want at the classroom level.”

(City, 2009, p 32)

– There is often a “lack of an agreed-upon definition of what high-quality instruction looks like.”

34

Education Has Not Been Scientific

• Education has been criticized for not being “scientific.”

• We do know a lot about how students learn various subjects.

• The problem is that this knowledge is not consistently applied in daily instructional practices.

35

Addressing #1—Lack of Vision• “Develops, with colleagues who have to work

together on school improvement, a shared understanding of what they mean by effective instruction.”

(City, 2009, p 10)

• Build vision of effective instruction upon knowledge of how students learn the subject– Most instructional practices fail to effectively apply

what research tells up about how students learn a given subject.

Examining Research

37

Reading Research Converges• Reading—50 years of research• Effective reading instruction requires a

balanced blend of:• Phonemic awareness• Decoding• Vocabulary development• Reading fluency, including oral reading skills• Reading comprehension strategies

• Any single approach is inadequate

38

Math Research Converges

• National Math Panel Report

• Effective mathematics instruction requires a balanced blend of:

• Computational fluency

• Concept development

• Problem solving

• Any single approach is inadequate

39

What About Science?

40

Science Is Different FromLanguage Arts and Math

• Language Arts (& Reading) and Math– Are skills created by people– Involves learning established conventions

• Science– Understanding how the world works– How to create new knowledge– Living in the world creates a working understanding

which may or may not be scientifically valid

41

Importance of Learning Theory in Science

• Working knowledge is entrenched and difficult to overcome

• Sometimes called “Private Universe”• Key concepts must be internalized

– Sometimes called “Big Ideas” or “Enduring Understandings”

• Requires greater attention to the learning theory embodied in the instructional materials

42

Research On Effective Science Instruction is Also Converging

• Considerable evidence from research shows that instruction is most effective when it elicits students’ initial ideas, provides them with opportunities to confront those ideas, helps them formulate new ideas based on evidence, and encourages students to reflect upon how their ideas have evolved.

(Banilower, 2009)

43

Untrenching Their Private Universe

• Without these opportunities, students “may fail to grasp the new concepts and information that are taught, or they may learn them for purposes of a test but revert to their preconceptions outside the classroom”

(National Research Council, 2003, p. 14)

Therefore:• Effective science instruction must be

guided by what research tells us about how students learn science

• It is the duty of all science teacher to enact sound learning theory in science instruction

45

LASER Theory of Action• If teachers use research-based instructional practices,

materials, and assessments so that students:– Draw upon a deep foundation of usable knowledge within the

context of a conceptual framework to build scientific understanding– Are intellectually engaged and motivated– Reveal preconceptions and their initial reasoning– Use evidence to generate explanations– Communicate and critique their scientific ideas and the ideas of

others– Have full access to activities and sense-making discussions to

develop scientific understandings– Reflect on how personal understanding has changed over time

and recognize cognitive processes that lead to changes

Then:• The percent of students who meet or exceed the

science standards on the state and other science assessments will increase,

• Student participation and success in rigorous science courses K–12 will increase,

• The number of students who seek further studies in STEM content and STEM careers will increase, and

• The number of students who choose to do science-based activities out of school time will increase.

46

47

In Other WordsStudent science

achievement will increase if we consistently apply what

we know about how students learn science!

48

Common Pitfall #2• Lack of a Process

– Teachers attend professional development but then it is up to individuals to put the new knowledge and skills into practice

– “[Schools] don’t have a process for translating that knowledge systematically into practice.”

49

Addressing #2—Lack of Process• Establish a school-based collaborative process

that engages teachers in:– Regular structured observations of students for

evidence that the vision of how students learn the subject is being enacted

– Engage teachers in a process to make meaning from the observation experience to continuously improve instructional practices

• “Puts educators in a position of having to actively construct their own knowledge of effective instructional practice”

(City, 2009, p 10)

50

Common Pitfall #3• Atomized Culture

– “Most people in schools work in siloed cultures characterized by independence and autonomy.”

(City, 2009, p 62)

– Too often schools “are organizations that support the private practice of teachers” and the atomized culture.

– Atomized culture is one of the most significant barriers to school improvement.

51

Addressing #3—Atomized Culture• Deprivatize Practice

– “It is clear that closed classroom doors will not help us educate all students to high levels.”

– “Everyone is obligated to be knowledgeable about the common task of instructional improvement and everyone’s practice should be subject to scrutiny, critique, and improvement.”

– “We can do more together than we can individually to improve learning and teaching.”

(City, 2009, p 3)

52

Addressing #3—Atomized Culture

• Importance of Observation– “The only way to find out what

students are actually doing is to observe what they are doing.”

– “What predicts performance is what students are actually doing.”

(City, 2009, p 30)

Implications for Planning• Things to Remember

– The infrastructure and initial use training to support the use of a core curriculum of inquiry-based science instructional modules is necessary but not sufficient to raise student achievement in science

– Fill curriculum gaps

53

Implications for Planning• Embrace the LASER vision of effective

science learning experiences for students embodied in the LASER theory of action– Ensure all professional development focuses on

helping teachers enact that vision– Implementing the effective learning experiences

for students is the purpose of professional development

54

Implications for Planning• Establish an internal process for continuously

improving instruction that:– Provides time within the school day for participation– Has sufficient structure– Has creditably leadership– Provides access to content expertise– Deprivitizes practice– Involves observation– Strives for horizontal and vertical articulation

55

56

Questions ???Dave Weaver

RMC Research Corporation111 SW Columbia, Suite 1200

Portland, Oregon 97201dweaver@rmccorp.com

(800) 788–1887

Presentation is at:http://www.rmccorp.com/LASER/

57

“Must Read” List• City, E., Elmore, R., Fiarman, S., & Tietel, L.

(2009). Instructional rounds in education. Cambridge, MA: Harvard Education Press.

• Banilower, E., Cohen, K., Pasley, J., & Weiss, I. (2008). Effective science instruction: What does research tell us? Portsmouth, NH: RMC Research Corporation, Center on Instruction.

• National Research Council. (2003). How people learn: Brain, mind, experience, and school. J. D. Bransford, A. L. Brown, & R. R. Cocking (Eds.). Washington, DC: National Academy Press.