impact of a school district's science reform effort on the achievement and attitudes of third-...

JOURNAL OF RESEARCH IN SCIENCE TEACHING VOL. 41, NO. 8, PP. 771–790 (2004)

Impact of a School District’s Science Reform Effort on the Achievementand Attitudes of Third- and Fourth-Grade Students

James A. Shymansky,1 Larry D. Yore,2 John O. Anderson2

1Regional Institute for Science Education, Suite 7, RCEW Building, 8001 Natural Bridge Road,

University of Missouri, St. Louis, Missouri 53121-4499

2Faculty of Education, University of Victoria, Victoria, British Columbia, Canada V8W 3N4

Received 25 November 2003; Accepted 6 February 2004

Abstract: This article is about one school district’s effort to reform its elementary science curriculum

through a program of professional development called Science, Parents, Activities and Literature (Science

PALs). The differential exposure of the district’s K–6 teachers to Science PALs and differences in how well

teachers implemented Science PALs-type inquiry strategies allowed us to conduct a quasi-experimental

study of the impact of Science PALs on student achievement and attitudes. We measured achievement with

an instrument based on items taken from the Third International Mathematics and Science Study (TIMSS;

International Association for the Evaluation of Educational Achievement, 1997) and selected attitudes

about science with the Student Perceptions of Classroom Climate (SPOCC; Yore et al., 1998), an instrument

that we designed. Our analyses of student attitude scores as a function of years of teacher participation

in Science PALs and supervisor’s rating of a teacher’s implementation of the project’s instructional

approaches showed a significant overall positive impact on student attitudes toward school science. Student

TIMSS scores on multiple-choice items or constructed-response items did not improve significantly when

analyzed by the number of years a student’s teacher was involved in the Science PALs effort or by the

supervisor’s rating of that implementation. We found no significant differences in attitude or achievement

scores among students taught by a series of teachers rated high, medium, or low in quality of imple-

mentation by the district’s science supervisor. We discuss possible explanations for the lack of clear and

positive connections between Science PALs and student performance in light of the increased focus on

accountability in reform projects. � 2004 Wiley Periodicals, Inc. J Res Sci Teach 41: 771–790, 2004

Professional development of teachers in the areas of science and mathematics continues to

receive strong support from the National Science Foundation (NSF) because of persisting

evidence that large numbers of untrained and undertrained teachers still enter the classroom each

year (National Commission on Teaching and America’s Future, 1996). The NSF support,

Correspondence to: J.A. Shymansky; E-mail: [email protected]

DOI 10.1002/tea.20025

Published online 25 August 2004 in Wiley InterScience (www.interscience.wiley.com).

� 2004 Wiley Periodicals, Inc.

however, is not without its critics. In their review of teacher-enhancement programs, Frechtling,

Sharp, Carey, and Vaden-Kierman (1995) noted that ‘‘most program evaluations either ignore

student achievement or provide unconvincing . . . anecdotal teacher reports of positive student

outcomes’’ (p. 18). There is strong pressure on those who provide professional development

to show evidence that the efforts are impacting student achievement (Education Commission of

the States, 1997). And as Loucks-Horsley and Matsumoto (1999) stated in their review of the

professional-development research, despite this mounting pressure, ‘‘there is relatively little

research addressing this connection’’ (p. 258).

Slavin (2003) explained this lack of ‘‘scientifically based research’’ on student achievement:

‘‘In addition to what a teacher does in the classroom, student achievement is affected by other

things such as socioeconomic status, school administrators’ and parents’ knowledge and

commitment, school district and state policies regarding standards and testing, . . . and the list goes

on’’ (p. 12). These complexities make the design and execution of true experimental studies that

can isolate and measure the specific effects of professional development on student achievement

both difficult and expensive. But the question of accountability is not likely to disappear just

because persuasive evidence is difficult and expensive to collect.

This article is about one school district’s effort to reform its elementary science curriculum

and the effect of that reform on student achievement and attitudes. We measured achievement

with an instrument based on items taken from the Third International Mathematics and Science

Study (TIMSS; International Association for the Evaluation of Educational Achievement,

1997) and selected attitudes about science with the Student Perceptions of Classroom Climate

(SPOCC; Yore et al., 1998), an instrument that we designed. The district already had an extensive

hands-on curriculum in which science kits were circulated among teachers during the school year.

The curriculum was supported by a half-time science supervisor and a full-time materials dis-

tribution and maintenance center employee. Each year, the science supervisor provided about

2 days of professional development that focused mainly on activity mechanics and little on any

theoretical underpinnings for inquiry-based learning. Students and teachers in the district seemed

to enjoy doing the science activities contained in the kits; however, despite the support of the

district and the enthusiasm for doing activities, most teachers admitted to having little under-

standing of the science ideas being explored in the kits or the rationale for doing inquiry-based

instruction and, more important, they felt they had no real evidence on how their science

instruction was impacting students. There was a strong sense among teachers, administrators, and

parents that their K–6 science program was good, but they also felt that students could be learning

even more.

Working with university experts and with special funding from the NSF and Howard Hughes

Medical Foundation, the district launched a reform effort called ‘‘Science Parents, Activities and

Literature’’ (Science PALs). The goal of Science PALs was simple: Increase the kit-based

program’s capacity to promote student conceptual growth and positive attitudes toward science

without dampening the enthusiasm that the kits seemed to generate among teachers, students, and

parents. Science PALs set out to provide a sustained program of professional development focused

on a constructivist theory-based, four-part teaching strategy to support inquiry science learning,

ways to involve parents in the science instruction, and strategies to connect science to other areas

in the curriculum. Though the summer workshop setting served as the primary vehicle in Science

PALs, the workshops were anything but traditional, utilizing ‘‘reform structures’’ described by

Garet, Porter, Desimone, Birman, and Yoon (2001). Special K–6 teacher ‘‘advocates’’ and Grades

7–12 ‘‘science partners’’ served as mentors for kit-focused collaborative groups. University

science and science education experts worked, for the most part, through the advocate–mentor

teams in supporting roles rather than as workshop instructors.

772 SHYMANSKY, YORE, AND ANDERSON

The need to initially and continually elicit student’s ideas and then challenge and extend those

ideas is the cornerstone of all varieties of constructivism. In Science PALs, two unique strategies

for eliciting student ideas were emphasized: (a) using children’s literature with story lines

containing embedded science ideas (both scientifically valid and invalid ideas) and (b) using

parents to help identify student ideas. During the funded phase of the reform (Years 1–4) effort,

73% of all elementary science teachers in the district participated in 1 to 3 years of Science PALs

professional development while 27% declined any involvement. All district teachers, however,

continued to use the same kits of science materials (discussed later). Moreover, we recognized that

both participating and nonparticipating teachers were using Science PALs-type inquiry strategies

to varying degrees, though none had received any formal instruction in constructivist learning

theory or its related teaching models. The differential exposure to Science PALs professional

development and differential utilization of Science PALs-type inquiry strategies combined with

the use of common science kits, however, provided an opportunity to conduct a quasi-

experimental study focusing on the following research questions:

� How does number of years of teacher exposure to Science PALs professional-

development activities impact student attitudes toward school science, attitudes about

careers in science, and achievement in science?

� Does level of teacher exposure to or quality of utilization of strategies learned in Science

PALs professional-development activities differentially affect boys’ and girls’ attitudes

toward school science, attitudes about careers in science, and achievement in science?

� How does the quality of teacher utilization of Science PALs professional-development

activities during the current year of school affect student attitudes toward school science,

attitudes about careers in science, and achievement in science?

� How does the cumulative quality of teacher utilization of Science PALs professional-

development activities over 3 years of school affect student attitudes toward school

science, attitudes about careers in science, and achievement in science?

This study adds to the professional development and reform literature in two important ways

identified by Loucks-Horsley and Matsumoto (1999): (a) It provides data linking a sustained

program of professional development (Science PALs) to student achievement and selected atti-

tudes, and (b) it addresses the problem of disentangling the complex professional development–

student performance connection by accounting for differences in exposure to and quality of

implementation of practices and strategies targeted in (Science PALs) professional development.

Background

Several documents provided guidance and underpinning for the structure and content of the

Science PALs reform effort: Science for All Americans (American Association for the

Advancement of Science, 1990), Benchmarks for Science Literacy (American Association for

the Advancement of Science, 1993), the National Science Education Standards (National

Research Council, 1996), and the Report of the National Commission on Teaching and America’s

Future (Darling-Hammond, 1996). Collectively, the documents provide a vision of what science

should be taught and how science should be taught.

Concerning ‘‘what science should be taught,’’ reform-document authors speak of science

literacy, being able to engage intelligently in public discourse and debate about issues that involve

science and technology, and sharing in the excitement, personal fulfillment, and positive feelings

that can come from understanding and learning about the natural world (National Research

IMPACT OF A SCHOOL DISTRICT’S SCIENCE REFORM 773

Council, 1996). Authors of Science for All Americans, Benchmarks for Science Literacy, and the

National Science Education Standards contend that all students should be challenged to develop

an understanding of a few powerful, unifying science ideas and that inquiry should be the

foundation for learning science. Norris and Phillips (2003) described this duality of science

literacy as being composed of a fundamental sense involving the use of language in and about

science and a derived sense of understanding the big ideas of science.

On the issue of ‘‘how science should be taught,’’ reform-document authors speak of teachers

having theoretical and practical knowledge about science, learning, and science teaching

(National Research Council, 1996). This knowledge includes being able to plan, design, and

manage curriculum that promotes the procedures and the intellectual rigor of scientific inquiry and

to assess their own teaching and student learning. The vision consistently described in the

collective reform documents is of science teaching that engages all students in a quest for science

literacy involving the abilities, critical thinking, and habits-of-mind to construct understanding of

the big ideas and unifying concepts of science, and the communications to share with and persuade

other people about these ideas (Ford, Yore, & Anthony, 1997).

When these changing emphases in teaching are considered in the context of science as inquiry

and technology as design and the epistemology described by the nature of science as a way of

knowing and seeking the best possible explanations based on empirical standards, logical

arguments, and skepticism, it becomes apparent that an ‘‘interactive-constructivist’’ perspective is

supported by the National Science Education Standards (National Research Council, 1996). An

interactive-constructive model utilizes an ecology metaphor to illustrate learning in which

dynamic interactions of prior knowledge, concurrent sensory experiences, belief systems, and

other people in a sociocultural context lead to multiple interpretations that are verified against

evidence and privately integrated into a person’s knowledge network (Shymansky et al., 1997).

Knowledge, in this model, is perceived as individualistic conceptions of reality that have been

verified by the epistemic traditions of a community of learners.

Reform efforts of the past 15 years and the resultant ‘‘standards’’ documents have succeeded

in defining what and how science should be taught, but the science education community

continues to struggle with programs of professional development that will establish a clear link

to student outcomes. Loucks-Horsley and Matsumoto (1999) proposed a model of ‘‘influences

on the relationship between professional development and student learning’’ (p. 260) that iden-

tifies school, cultural, parental, policy, and leadership factors embedded in state and national

contexts as critical elements in the teaching–learning equation. To this equation we would add

‘‘international’’ as a context factor, as evidenced by the worldwide interest generated by the

TIMSS results.

Compounding the problem of doing research that directly connects professional development

to student achievement is the fact that effects of professional development on classroom practice

do not often manifest themselves immediately. Shields, Marsh, and Adelman (1998) and Weiss,

Montgomery, Ridgway, and Bond (1998) suggested that the depth of teacher change is directly

related to the duration of professional development efforts. Supovitz and Turner (2000) found that

teachers experiencing less than 40 hr of professional development often reported inquiry-based

practices and investigative classroom cultures that were less positive than teachers receiving no

professional development at all. They found that it was not until teachers received more than about

80 hr of professional development did their reported inquiry practices and investigative culture

significantly improve. Assuming that it would take an extended time for students to adapt to and

resonate with changes in classroom practice and culture, short-term research studies seeking to

establish a link between improved inquiry practices and student achievement would almost

certainly fail to reveal the link.


Theoretical Framework

The teacher-enhancement approach utilized in Science PALs was based on a situated learning

model of professional development relying heavily on authentic teacher work and professional

mentorship (Lavoie & Roth, 2001). The authentic teacher work provided opportunity and context

for preparing, adapting, teaching, reflecting, and revising a specific science unit associated with

the participating teacher’s professional responsibility and grade-level assignment. A common

difficulty with changing teachers’ vision of effective science teaching and facilitating the

associated practice in their classrooms is their lack of direct experience with the desired approach,

awareness of the instructional resources, and pedagogical-content knowledge associated with the

science unit (Smith, 1999; Zembal, Starr, & Krajcik, 1999). Science PALs addressed these

problems by using the desired teaching approach—interactive-constructivist science inquiry

teaching—as the instructional approach for the professional-development activities. By using the

science kit to be taught as the conceptual environment and physical context for grade-specific

activities and the prior knowledge, curricular priorities, and pedagogical strengths of the generalist

elementary teachers as the starting point for instruction and for selecting project goals, the

fundamental and derived senses of science literacy, interactive-constructivist teaching common to

language arts, mathematics, and social studies reforms, and cross-curricular connections were

woven prominently into all aspects of Science PALs.

Frequently in professional-development projects, teachers are introduced to innovations and

concepts but are not given experience in how to incorporate and apply these new ideas to their

elementary classrooms. The transfer problems were reduced in Science PALS by establishing a

mentorship among a participating teachers, an expert pedagogical lead elementary teacher, and

a science-content-expert secondary science teacher or project staff member. The mentorship

provided a supportive climate for the implementation of the science unit and addressed day-to-

day logistical problems, content difficulties, teaching issues, assessment concerns, and cross-

curricular connections in the context of real classroom science teaching as the problems arose,

providing just-in-time professional development on an individualized basis. The general pattern

of the mentorship involved preparing the science unit during a summer workshop, supervisory

support during the teaching of the unit in the late fall or early spring, and guidance during the

reflection on the teaching experience and the revision of the science unit based on the evaluation

of the science instruction experience in late spring. This cycle was repeated in each year of

participation with a different science unit.

Setting

The Science PALs project was funded as a teacher-enhancement partnership between a level-

one research university and a single school district. In the 5 years of the project (Years 1–4 were

externally funded, and Year 5 was supported by the school district.), 238 K–6 teachers and 15

Grades 7–12 teachers from the school district participated in summer and school-year inservice

activities. In addition to the teachers, approximately 3,400 parents participated in special training

sessions designed to integrate them into the K–6 science program. Across the 5 years of Science

PALs, teachers received an average of 110 hr of professional development designed to enhance

their science content and pedagogical-content knowledge.

At the beginning of the Science PALs Project, the participating school district had a kit-based

elementary school science program that contained exemplary NSF-supported materials, such as

the Full Option Science System (FOSS; Carolina Biological), Science and Technology for

Children (STC; National Science Resources Center), and INSIGHTS (Educational Development


Center). The kits were delivered to the teacher on a rotating basis with minimal professional

development.

The first year of the Science PALs Project began with 16 elementary school teachers

designated as science advocates—one from each elementary school in the district. These teachers

were selected in part for their willingness to serve as science leaders in their schools as well as their

interest in participating in the teacher enhancement project. The science advocates began the

project by attending a special problem-centered summer workshop similar to the Focus on

Children’s Ideas in Science project (FOCIS; Shymansky et al., 1993). The FOCIS project utilized

middle school science teachers’ interest in children’s misconceptions and their sincere desire to

promote conceptual change in their students as an authentic problem focus for the summer

workshop and multiyear collaboration with a science-content mentor. The focus on children’s

ideas served as the ‘‘straw man’’ since enhancement of the teachers’ science-content knowledge

and pedagogical-content knowledge were the actual goals of the project. The FOCIS project was

effective in achieving meaningful science and science pedagogical learning among the middle

school teachers on science topics of their choice.

The Science PALs workshops in that first year were designed to help teachers explore selected

curriculum units (NSF-supported versions) and activities using students’ ideas again as the ‘‘straw

man.’’ The workshops matched science-content consultants with small groups of advocates to

explore the science concepts in specific units and to promote an interactive-constructivist teaching

strategy among the teachers. In the workshop and the ensuing school-year inservice sessions,

various strategies were employed to have the science advocates articulate their alternative

frameworks for the science concepts related to the school district’s science units, and additional

extension activities to challenge these understandings were implemented. The ultimate objective

was to address the advocates’ personal misconceptions and have them rethink their under-

standings to develop more accurate scientific conceptions critical to teaching the unit. These

science advocates then supplemented the specific FOSS, STC, and INSIGHTS units with under-

standings of the science reforms, misconception literature, additional science activities, children’s

literature, and interdisciplinary connections to produce teacher resource binders (TRBs) for each

science unit.

The 16 science advocates field tested the enriched units (field-test versions) in their own

classrooms in the fall and attended three 1-day workshops during and after teaching the units.

The field-test experiences were shared with colleagues and science-content consultants to

further clarify science understandings and explore other activities to challenge student mis-

conceptions uncovered while teaching the unit. These insights were used to revise the TRBs for

each science unit (final version) and to develop home science-activity bags. The activity bags

consisted of a children’s literature selection related to the central science topic of the unit, a

simple science equipment, and a parent interview and activity guide. Parents and children read

the story together and explored various science challenges in the story as they occurred, using

the activity guide and equipment provided in the activity bags. Feedback from parents was

used to make adjustments to the science instruction that more accurately reflected their students’

prior knowledge. Parent orientation meetings were developed to introduce parents to the Science

PALs project and activity bags. A Science PALs project newsletter was published to keep the

parent community informed about the project’s progress and to maintain contact with students’

families.

The cascading leadership design of Science PALs involved a progression of participating

teachers and an evolution of their specific leadership roles. Early months of the project focused on

recruiting and working with the 16 science advocates. Twelve of the original advocates continued

to serve in the advocate capacity after external funding for the project ceased. Year-2 activities


focused on recruiting and working with 24 lead teachers to complement and share leadership

responsibilities with the advocates in a school. Year-3 activities focused on 37 additional teachers

recruited as a cohort to join the 40 advocates and lead teachers. In the final year (Year 4) of funding,

the advocate, lead teacher, and teacher group consisted of 140 teachers, bringing the total active

K–6 teacher cohort in PALs across the 4 funded years of the project to 195 different teachers. This

number represents about 73% of the 267 K–6 teachers in the school district and about 90% of the

216 who taught science on a regular basis. The cascading leadership model used meant that

advocates and lead teachers progressively assumed greater responsibility for the summer

workshops, professional development activities, and science decisions. Table 1 shows the layout

of a typical year of professional development in Science PALs.

Research Design

The study that we conducted to determine the impact of Science PALs on student performance

is best described as a posttest only, nonrandomized, comparative-group quasi-experiment. Years

of teacher participation or exposure to the Science PALs professional-development activities and

the level of teachers’ utilization of Science PALs strategies were used as the main independent

variables in our study. The exposure variable represents nothing more than amount of teacher

involvement in the summer workshop and school-year inservice activities in a 12-month period. It

was chosen for study because of its prevalence in the reform literature. The utilization variable

represents the effectiveness with which a teacher used student ideas and selected children’s

literature and parent partners to plan and implement instructional activities promoted in the

professional development/reform effort. The utilization variable was chosen for study because it

simultaneously addresses the problems of fidelity (participating teachers not embracing and

implementing the selected reform ideas and strategies) and of history and contamination

(nonparticipating teachers using the same selected reform ideas and strategies that were learned

previously or indirectly from participating teachers).

The science supervisor’s ratings of the teachers’ utilization of Science PALs strategies

(described later) allowed us to examine utilization by the students’ current teacher as well as

Table 1

A 1-year cycle of Science Pals professional development activities

School Year Activity Focus

May 2-day retreat of science advocates, project staff, and district administrators toclarify project goals and critical features

June–July Summer workshop—select and explore target science; develop cross-curricularconnections, parent bags, and TRBs

August Share unit plans and organize implementation; plan parent orientation meetings andin-service schedules

September Parent orientation: Conduct parent meetings, distribute activity bags for first fallunit, collect parent activity data

October–December Explore additional cross-curricular connections and alternative assessments; workon new science units for fall and spring

January–April Follow-up on parent activity data, assessment ideas, children’s literature, sciencecontent, applications of technology, etc.

May Science advocates’ retreat—leadership responsibilitiesJune–July Introduce new teachers and develop new science units

Note. TRBs¼ teacher resource binders.


another variable: the cumulative effect of Science PALs on student performance over a 3-year

period. The cumulative-effect factor was determined by combining the science supervisor ratings

of a given student’s current-year teacher and his or her teachers of the previous 2 years.

Selected student attitudes toward school science and student achievement were chosen as the

dependent variables for study. The attitude measures that we report focused on students’ percep-

tions of the nature of science and interest in a possible career in science. Student achievement was

measured with multiple forms developed from released science items from the TIMSS for Grades

3/4 (International Association for the Evaluation of Educational Achievement, 1997). Student

gender was added as a secondary independent variable to examine any differential effects of the

Science PALs strategies on girls and boys.

Instruments

Teacher Ratings

The school-district science supervisor determined the degree to which teachers utilized

interactive-constructivist teaching methods generally and Science PALs strategies specifically in

their science teaching. All K–6 teachers in the district were rated regardless of how or where these

attributes were acquired. The science supervisor rated each teacher (Science PALs and Non-

Science PALs) on a five-dimension rubric developed to assess the unique features of the Science

PALs approach and the global features of an interactive-constructivist approach. The rubric

required the supervisor to assess the degree of compliance on each dimension (1¼ very weak,

2¼weak, 3¼ satisfactory, 4¼ strong, 5¼ very strong). Thus, an individual teacher’s Science

PALs rating could range from 1 to 5 on the overall rating (rubric Item 5) or 4 to 20 on the sum of the

specific Science PALs strategies (rubric Items 1–4) as shown in Table 2.

The substantive validity, external validity, structural validity, and reliability of the rubric were

established by a series of inquiries (Messick, 1989). First, the substantive validity was considered

by insuring that the dimensions in the rating scale matched the theoretical and practical

assumptions and goals of the project and that the Science PALs features were associated with

interactive-constructivist science teaching. The results of Dimensions 1 to 4 (Science PALs

features) were correlated with Dimension 5 (global indicator of interactive-constructivist

teaching) as individual pairs (Correlations of the Year-3 ratings were 0.68–0.95, and correlations

of the Year-4 ratings were 0.78–0.95.) and as a unified cluster (Correlation of the Year-3 ratings

was 0.94, and correlation of the Year-4 ratings was 0.96.) The external validity was explored by

t-test analyses of 128 teachers’ ratings over a complete Science PALs cycle. Comparisons of

the clustered dimensions (Science PALs approach) and the global dimension (interactive-

Table 2

Science PALs rating rubric

Rating Science PALs Teaching Practice

1–5 Use of strategies to access and utilize information on student ideas in planning instruction1–5 Use of strategies to challenge student ideas and to have them reflect on and integrate those ideas

into their thinking1–5 Use of strategies that routinely and continuously use children’s literature and personal

experiences as context for learning science1–5 Use of strategies that promote ongoing, substantive parent involvement in the science

instruction1–5 Overall rating as a constructivist teacher, as defined in the goals of the Science PALs project


constructivist approach) revealed significant predicted improvements in the ratings, clustered

dimension: t¼ 5.0, p< 0.001, global dimension: t¼ 4.2, p< 0.001.

The structural validity was checked by a series of factor analyses utilizing one- and two-factor

solutions. Both solutions were supportive (component loadings of 0.80–0.97), indicating that the

five dimensions were acting as a unified factor or that the first four dimensions could be considered

a factor, with Dimension 5 acting as a separate, single-item factor. Reliability was supported by the

correlation results and by a rate-rerate analysis of a random sample of 12 teachers. Measures

of internal consistency for the four-dimension cluster were 0.96 (Year-3 ratings) and 0.97 (Year-4

ratings). The science supervisor’s rating consistency was explored by asking her to rate

216 teachers who teach science in the 16 elementary schools. One week later, a 5% random sample

of 12 teachers was rerated by the science supervisor on the ploy that their rating results were lost.

The correlation of these paired ratings was 0.95.

Student Achievement

The released items from the TIMSS for Grades 3/4 were collected into six tests to assess

science achievement (International Association for the Evaluation of Educational Achievement,

1997). Each form of the science achievement test consisted of 32 items: 25 multiple-choice items

and 7 constructed-response items (short-answer and extended-response) that were categorically

aligned by a representative group of school-district teachers who judged the items as being

relevant to the science topics addressed in the district’s K–6 science framework (Webb, 1999). All

forms of the test had nine common multiple-choice items and three common constructed-response

items. Various combinations of the forms had other common constructed-response items. The

validity was assumed to be reasonable since the items were selected from a pool of items used in a

major international assessment project (International Association for the Evaluation of Educa-

tional Achievement, 1997). The reliability of the six forms was measured by the internal

consistencies of each form. The consistency values ranged from 0.73 to 0.77.

The six forms were randomly distributed across students in each classroom tested so that

some students were assessed by each of the six forms of the TIMSS test in every teacher’s

classroom. Statistical analysis of the results on the common items across the six subgroups taking

the various forms of the achievement test revealed no significant differences. Based on this

evidence, it was assumed that the test forms were not biased or of different difficulties and that the

results across all forms of the test could be combined using standard scores (Z¼ 50þ 10z score,

SD¼ 10SDz) to adjust results for the uncommon composition. This allowed us to examine the

Z scores for the multiple-choice and constructed-response subscales (short-answer and extended-

response) as measures of lower level and higher level knowledge to further inform the research

questions.

Multiple-choice items on the TIMSS were optically scanned and scored against a master key.

Constructed-response items were hand scored by an independent expert rater using scoring rubrics

developed by the TIMSS project staff. The full set of one constructed-response item was scored

before moving to the next item. A random sample of 10 responses per 100 responses for each of the

constructed-response items was double scored to monitor scorer reliability. Rater agreement for

the constructed-response items across all items averaged 0.95.

Student Attitudes

An instrument containing Likert-type items designed to assess students’ agreement, absence

of opinion, or disagreement on a five-position response scale to statements describing students’


affective stance toward science and science careers was used to measure students’ perceptions and

attitudes (Dunkhase, Hand, Shymansky, & Yore, 1997; Shymansky, Yore, Dunkhase, & Hand,

1998; Yore et al., 1998). Validity and reliability of this attitude and awareness survey was explored

using expert analysis, factor analyses, and internal consistency. The construct validity of the

instrument was investigated by having experts examine the items selected from established item

pools or constructed by science educators for the project. Factor analyses were conducted on the

Year-2 results from 722 students in Grades 3 and 4 taking the original pool of items. The final

version of the instrument retained only those items that had a > 0.30 loading value. The items

retained were analyzed again to insure that the resulting factors matched the design features of the

instrument. The two factors retained were the attitudes toward school science (nine items with a

total score range of 9–45) and awareness of science careers (three items with a total score range of

3–15). Internal consistency was 0.79 for the combined scales, 0.74 for the attitudes toward school

science subscale, and 0.72 for the science careers subscale. The same instrument administered to

456 students the next year revealed internal consistencies of 0.75, 0.74, and 0.69 for the respective

combined scales and the two subscales on which our study focused.

Results

The project staff and the school district teachers, parents, and administrators were interested

in learning how the trait variable of years of Science PALs professional development or exposure

impacted student attitudes and achievement in science. Table 3 contains descriptive statistics for

the analyses exploring the effect of years of professional development on student attitudes and

achievement of students in Grades 3 and 4 without regard to student gender.

Of interest in Table 3 are (a) the decline in student attitudes toward school science coupled

with an increased interest in science careers among students in classrooms of teachers with 2 years

of PALs training and (b) the drop in performance on the constructed-response portion of the

TIMSS test among students in classes of teachers with 1 to 2 years of PALs training. On all

measures, the student scores showed a tendency to increase in some cases, but also to decrease in

other cases as a function of teacher exposure to Science PALs professional development. Analyses

of variance and pairwise comparisons of the student scores showed that these undulations were

significant in several instances, as shown in Table 4.

We also analyzed the data to determine any differential effects of years of Science PALs

professional development on girls’ and boys’ achievement and attitudes. Boys significantly

(p� 0.05) outscored girls on both the TIMSS multiple-choice items and the total TIMSS test

regardless of the number of years of Science PALs professional development experienced by their

Table 3

Summary statistics for Grade 3/4 student attitudes and achievement (Z score) as a function of teacher years

of Science PALs professional development

Years

Science Careers TIMSS-MC TIMSS-CR TIMSS-TS

n M SD n M SD n M SD n M SD n M SD

0 154 27.0 3.0 159 7.9 2.2 241 51.3 9.6 225 51.8 8.9 208 51.3 9.61 185 27.7 2.6 185 7.9 2.0 308 50.2 9.7 294 49.9 10.2 270 49.4 9.92 98 27.3 3.2 98 8.6 2.4 112 50.1 10.5 123 47.9 11.0 97 50.0 10.43 139 27.9 3.0 139 7.6 2.5 203 49.5 10.3 189 50.7 9.3 172 49.6 10.1

Note. TIMSS-MC¼Third International Mathematics and Science Study–Multiple Choice; CR¼Constructed Response;

TS¼Teacher Scores.


teacher. There were no significant (p> 0.05) differences between boys and girls on the TIMSS

constructed-response items or either of the attitude scales. Descriptive statistics for the student

gender analyses are presented in Table 5.

Analysis of the differential effects of a given teacher’s utilization of the Science PALs

teaching approaches on his or her students’ attitudes and achievement involved the science

supervisor’s ratings. Two ‘‘quality of implementation’’ scores were created for each teacher in

each of the last 3 years of the project: (a) a ‘‘utilization of Science PALs strategies’’ score

represented by the combined science supervisor’s ratings of a teacher’s use of student ideas, level

of parental involvement in the science classroom, and use of children’s literature in the science

instruction (Items 1–4, Table 2) and (b) a ‘‘utilization of constructivist strategies’’ score repre-

sented by the science supervisor’s rating of a teacher’s overall use of interactive-constructivist

strategies (Item 5, Table 2). Composite ratings of the Science PALs strategies of 16 or greater were

considered ‘‘high,’’ composite ratings of 9 to 15 ‘‘medium,’’ and composite ratings of 8 or less

‘‘low’’ for analyses (Table 6). Overall ratings of the interactive-constructivist strategies of 4 or

5 were considered ‘‘high’’ ratings, of 3 ‘‘medium,’’ and ratings of 2 and 1 ‘‘low’’ for analyses

(Table 7).

Analyses of variance were conducted on the students’ attitudes, awareness, and achievement

for the supervisor’s composite and overall ratings of their current teacher’s science instruction.

These analyses revealed significant main effects for the supervisor’s composite ratings of

implemented Science PALs features, F¼ 7.20, df¼ 2, 603, p< 0.001, and for supervisor’s overall

rating of constructivist teaching, F¼ 9.21, df¼ 2, 603, p< 0.001, on students’ attitude toward

school science. No other significant main effects were found for attitudes toward science careers,

TIMMS multiple-choice items, TIMSS constructed-response items, and TIMSS total test scores.

Pairwise comparisons of the significant main effects for attitudes toward school science revealed

significant (p< 0.05) differences between all pairs for the ratings on ‘‘current teacher’s utilization

of Science PALs strategies,’’ Low<Medium (effect size, h¼ 0.20), Low<High (h¼ 0.56), and

Medium<High (h¼ 0.36); for ratings of ‘‘current teacher’s utilization of overall constructivist

strategies,’’ Low<Medium (h¼ 0.27), Low<High (h¼ 0.54), and Medium<High (h¼ 0.27).

The last analysis that we performed examined the long-term impact of Science PALs

professional development on student achievement and attitudes. We were able to study this impact

Table 4

One-way analyses of variance and the significant pairwise comparisons of Grade 3/4 student achievement

and attitudes as a function of years of Science PALs professional development

Student MeasureMain Effect

p Value Pairwise Comparisons

Attitude–School Science 0.037 Year 1>Year 0; Years 3>Year 0Attitude–Science Careers 0.013 Year 2>Years 0, 1, 3TIMSS-CR (Constructed Response) 0.004 Year 0>Years 1, 2; Years 3>Years 2

Table 5

Summary of Z-score statistics for Grade 3/4 boys’ and girls’ attitudes and achievement across all years of

Science PALs professional development

Gender



Male 293 27.7 3.0 293 8.0 2.5 465 51.3 9.9 443 50.2 9.9 399 51.1 10.0Female 290 27.3 2.9 290 7.9 2.0 448 48.9 9.7 434 50.0 9.9 388 48.9 9.8


by summing the science supervisor’s ratings of a given student’s current-year teacher and the

student’s teachers from the previous 2 years—for Grade-3 students, their current-year teacher and

their Grade-1 and Grade-2 teacher; for Grade-4 students, their current-year teacher and their

Grade-2 and Grade-3 teacher. The supervisor’s ratings of three teachers were combined to yield a

composite teacher-utilization score for each student on the Science PALs subscale [the sum of the

ratings of Items 1–4 on the Science PALs rating rubric (see Table 2) and on the ‘‘overall

constructivist’’ rating (Item 5 on the Science PALs rating rubric)]. The 3-year combined score

could range from 15 to 60 on the Science PALs subscale and from 3 to 15 on the ‘‘overall

constructivist’’ item.

Students taught by a sequence of teachers across a 3-year period with a combined Science

PALs utilization rating of 46 to 60 were considered as having a ‘‘high’’ quality Science PALs

learning experience; students taught by teachers with a combined utilization rating of 30 to 45

were considered as having an experience of ‘‘medium’’ quality; and students taught by teachers

with a combined utilization rating of 12 to 29 were considered as having an experience of ‘‘low’’

quality. In similar fashion, students taught by a sequence of teachers across a 3-year period with a

combined rating of 12 to 15 were considered as having an ‘‘overall constructivist’’ experience of

‘‘high’’ quality; 7 to 11 as ‘‘medium’’ quality; and 3 to 6 as ‘‘low’’ quality.

When we analyzed the student achievement and attitude data as a function of the quality of

the Science PALs teaching experienced by students across 3 years and the quality of the overall

constructivist teaching experienced by students across 3 years, we found no significant (p> 0.05)

differences. The descriptive statistics for the quality of Science PALs and the quality of the overall

constructivist teaching experienced by students across 3 years are presented in Tables 8 and 9,

respectively.

Discussion

Ongoing professional development of teachers is an accepted practice in the United States

that is supported by myriad statistics showing that many teachers are underprepared for their

Table 6

Summary statistics for Grade 3/4 student attitudes and achievement (Z score) as a function of the quality of

their current teacher’s utilization of Science PALs teaching strategies

Rating



Low 36 26.7 2.5 36 8.7 1.7 61 51.3 9.3 66 52.9 6.9 57 51.0 9.5Medium 377 27.2 3.1 377 7.9 2.4 541 49.9 9.7 521 49.9 10.3 466 49.7 9.9High 193 28.1 2.8 193 7.8 2.1 261 51.0 10.4 243 50.6 9.5 223 51.3 10.1

Table 7


their current teacher’s utilization of the overall constructivist approach

Ratings



Low 113 26.6 2.6 113 8.1 2.2 204 51.0 9.2 203 51.1 9.2 181 50.8 9.4Medium 253 27.3 3.2 253 7.8 2.3 341 49.8 9.9 333 49.6 10.6 296 49.4 10.1High 240 28.0 2.8 240 7.9 2.3 318 50.6 10.3 294 50.6 9.5 269 50.9 10.1


teaching assignments (National Commission on Teaching and America’s Future, 1996). Until

recently however, it was presumed that time and money invested in professional development

would lead to enhanced teacher quality and would eventually lead to improved student learning.

This link between professional development and student learning has gone essentially untested in

the area of science education for most of the last 40 years for two reasons beyond the cost and

difficulty of conducting such research: Professional-development efforts have rarely been applied

systemically and rarely over an extended period of time. The systemic factor is critical because

student learning and subsequent performance on high-stakes tests cannot be linked to any one

teacher, especially in the elementary school years. Elementary schools are complex systems of

interacting subsystems of students, teachers, and other factors such as parent involvement,

financial support for schools and programs, and administrative policies (Guskey & Sparks, 1996).

The extended time factor is critical because teacher behaviors learned and practiced over several

years and deficiencies in content and pedagogical-content knowledge cannot be undone or

meaningfully learned in a short period of time nor should the effects of improved teacher

background or teaching practices be expected to take hold immediately (Supovitz & Turner,

2000).

In our study of the effectiveness of Science PALs as a professional-development effort, we

were able to meet some of the challenges of trying to link the effort to student outcomes. Science

PALs was systemic. It targeted an entire teaching staff, eventually involving about 90% of the K–6

teachers who teach science. It also involved parents. Teachers developed special ‘‘take-home’’

activity bags and conducted yearly information and training sessions for parents—sessions that

enjoyed 90% parent participation (Shymansky, Yore, & Hand, 2000). The effort was sustained.

Teachers had the opportunity to participate in 40-hr summer workshops and 20 hr of professional

development during the school year for 3 consecutive years. System-wide, the mean number of

hours of professional development for the K–6 teaching staff was 1 full year, or 60 hr. The school

district enjoyed significant support for instructional materials. Science kits used by the K–6

teachers were housed in a separate facility, and prepared and maintained by a paraprofessional

staff. Finally, school-building principals, the superintendent, and the school-district board all

Table 8


Science PALs teaching experienced by students across 3 years

Quality



High 58 27.7 2.9 58 7.7 2.6 78 51.0 10.8 74 51.8 9.3 64 51.5 10.6Medium 458 27.5 2.8 458 8.0 2.2 431 51.1 9.6 422 51.1 9.4 377 51.3 9.5Low 34 27.9 3.3 34 8.0 2.7 94 50.7 9.1 89 52.5 8.3 81 50.7 8.9

Table 9


the overall constructivist teaching experienced by students across 3 years

Quality



High 239 27.5 2.8 239 7.7 2.4 266 57.3 9.9 252 51.2 9.1 227 51.6 9.7Medium 274 27.5 2.9 274 8.0 2.2 274 50.7 9.6 269 51.2 9.5 239 50.8 9.7Low 37 28.0 2.7 37 8.3 2.4 63 51.3 8.5 64 52.2 8.7 56 51.4 7.9


supported science as a curricular area and the Science PALs effort specifically carrying on with the

interactive-constructivist teaching approach, curriculum resource center, and the summer

workshops beyond the duration of the external funding. In short, it is reasonable to say that the

Science PALs professional-development effort appears to have been implemented in such a way

and under such conditions that our research questions about the impact of the effort on student

attitudes and achievement could be answered. So what did the analyses of our student attitude and

achievement data reveal?

Analyses of student attitude scores as a function of how many years their teachers participated

in the Science PALs professional development showed a significant overall positive effect in terms

of attitude toward school science (Year 3>Year 0), but a decline in attitudes about careers in

science after a significant spike in scores for students taught by teachers with 2 years of Science

PALs participation (Year 2>Years 0, 1, 3). These data suggest that students may have been

reacting positively to their teachers’ enhanced content knowledge, pedagogical-content knowl-

edge, and strategies learned in the Science PALs professional-development effort.

Analyses of student achievement data, however, tell a different story. Student scores on the

multiple-choice items of the TIMSS did not improve as a function of the number of years their

teachers were involved in the Science PALs effort and even declined slightly, though statistically

insignificantly. The same pattern was observed in the analysis of the total TIMSS score. On the

constructed-response items in the TIMSS, student scores actually declined in classrooms taught

by teachers participating in 1 to 2 years of Science PALs professional development before they

started to rebound (Year 0>Years 1, 2; Year 3>Year 2). This pattern is very similar to the one

described in Supovitz and Turner’s (2000) study. Their analyses showed that teachers’ reported

inquiry-based practice and investigative classroom climate declined initially and did not start

rebounding until the teachers had participated in about 80 hr of professional development.

Two years of Science PALs participation equals about 120 hr of professional development,

seemingly enough time for changes in classroom practices to take effect, but apparently not

enough time for those changes to show impact on student performance on multiple-choice or

constructed-response assessment items. The lack of improvement in the student scores on

multiple-choice items is not too great a concern, but the lack of improvement in the student-

generated response items is disappointing since the interactive-constructivist approach stressed

depth of understanding, which is the central focus of the higher level constructed-response items

and which may not be detected by the lower level multiple-choice items.

The results of analyses of student attitude and achievement as a function of years of

participation of teachers in Science PALs professional development were not surprising. There

were no significant differences in boys’ and girls’ scores on the attitude scales, but boys scored

significantly higher on the TIMSS multiple-choice items and the total TIMSS score regardless of

years of teacher participation in Science PALs. The nonsignificant difference in scores on the

TIMSS constructed-response items can be viewed as a positive gender result for the Science PALs

effort in light of the multiple-choice and total-score results.

The supervisor ratings of all teachers in the school district presented an opportunity for us to

link teachers individually and collectively to their students’ performance. Our idea was to look

specifically at how the ‘‘quality of implementation’’ of content knowledge, pedagogical-content

knowledge, and interactive-constructivist practices presented in the Science PALs professional-

development effort (quality as defined by the supervisor rating scale shown in Table 2) was related

to students’ attitudes and achievement. The ‘‘quality of implementation’’ allowed us to go beyond

an examination of the effect of ‘‘seat time’’ or mere ‘‘participation’’ in professional development

and to look at how teachers were actually using knowledge learned through Science PALs in their

classrooms. The extended term of our project further allowed us to study the impact that a series of


teachers might have on student attitudes and achievement: Would students who are taught by a

series of teachers rated ‘‘high’’ in quality of implementation by the district supervisor exhibit more

positive attitudes and higher achievement scores than students taught by a series of teachers

rated ‘‘medium’’ or ‘‘low’’ in quality of implementation? Unfortunately, we found no significant

differences.

The results of the students’ current teacher’s implementation of the Science PALs features

and the overall constructivist approach revealed the same significant main effects, consistencies,

and variations as described for years of Science PALs experiences. The analyses of the imple-

mentation quality of the students’ teachers over the last 3 years revealed no significant main effects

but similar patterns in the data as the other teacher factors.

Why did we find so few significant results in the various analyses performed? Teacher,

administrator, and parent reactions to the Science PALs program were all very positive (Yore,

Shymansky, & Anderson, in press). Everyone felt that the program was very successful, and

indeed, the program continues to receive strong support still today. Why do the attitude and

achievement data contradict or at least not support these positive feelings? The extended pattern of

argument encourages researchers to explicitly consider and rebut counterarguments and alterna-

tive interpretations for the claims, evidence, backings, or warrants presented. The consideration of

alternative claims and theoretical frames are often done during the research proposal, research

implementation, data collection, data interpretation, and report-writing stages. This process

continues naturally into the peer-review process of a manuscript where objective experts analyze

and evaluate the research reports and offer suggestions for conceptual and editorial improvements.

This has been the case during the coauthoring, review, and revision process for this article. There

has been divergent interpretations concerning the lack of success in documenting student-learning

effects as a function of professional development. We have offered our best interpretation of the

lack of significant linkages between the teacher-enhancement activities and student achievement

on a knowledge test and an attitude inventory, but there are several other contextual factors and

possible explanations that need to be considered. We describe them in the following paragraphs,

not necessarily in their order of importance or their validity as rival hypotheses.

The school district in which the Science PALs effort was implemented is not an ordinary

district. It is located in a medium-sized university town that boasts a relatively high concentration

of professional people with above-average education and an above-average median family

income. The K–6 teaching staff, indeed the full K–12 teaching staff, is fully certified and teaching

in field with most teachers of 5 or more years of experience holding master’s degrees and beyond.

The student dropout rate is very low, and the percentage who go on to postsecondary school is very

high. The school district ranks in the top one to three school districts in a state that ranks in the top

one to three states in standardized test scores and at or above the 90th percentile on the number of

National Merit Scholars in most years. All this information is a way of saying that students in the

school district do very well. In light of this, it is very likely that we encountered a kind of ‘‘ceiling

effect’’ in some of our test data, particularly on the achievement scores.

This ceiling-effect argument is supported by data in a report prepared for the local school

district administrative staff and school board (Table 10). The report shows the scores of students in

the school district compared to the national and international scores on the TIMSS. Grade 3 or

Grade 4 students in the school district equaled or outscored the national students on all but one of

the items and outscored the international students on all items tested. The comparisons among

district, national, and international performance are made crystal clear when attention is focused

on the constructed-response items. Since our research design and analyses focused on com-

parisons of teachers and their students within the school district, we were trying to find differences

among students from a group that was uniformly very strong overall. Had the Science PALs effort


been implemented in a school district where student achievement scores were distributed more

normally, the results might have been radically different from what we found.

Another possible explanation for the apparent lack of the Science PALs’ impact on student

attitude and achievement is the possibility of contamination of the untreated teachers in the district

(those who chose not to participate fully or not to participate at all). All teachers in the district had

access to the same science kits prior to the start of the Science PALs project and during the 4-year

term of the project. Teachers not participating in the Science PALs professional development did

not have direct access to the resource materials generated by participating teachers as part of the

professional-development activities (i.e., TRBs, special parent-activity bags, reading and writing

in science activities, assessments, cross-curricular activities, etc.). However, there was no attempt

to vigorously guard the special resource materials; nonparticipating teachers as well as those who

committed to less than the 60 hr of professional development each year had ample opportunity to

borrow and use materials and strategies from Science PALs colleagues. As a result, data analyses

based on level of involvement may have been compromised considerably.

Then there is the critical question of how well the TIMSS instruments that we used aligned

with the science units being taught. As mentioned earlier, the TIMSS items were aligned only

categorically, the most superficial level of alignment suggested by Webb (1997, 1999) and

currently being promoted by the American Association for the Advancement of Science (2001).

Not considered by teachers who were asked to align the TIMSS with the district’s science

curriculum was the range or depth of knowledge required by the various items or their demand in

terms of cognitive complexity (Impara, 2001). However, it was simply not feasible to train the

teachers to perform the more sophisticated alignment strategies even though a more rigorous

alignment of the TIMSS with the district’s K–6 science curriculum may well have produced a

fairer test of student achievement and more valid results for our study.

Table 10

How district Grades 3 and 4 students compare to U.S. and international students

on selected TIMSS items (average percent correct for Grade 3/4 students)

International U.S. District

Multiple Choice1. (moon-shine) 64/67 71/75 66/722. (temperature) 32/44 37/54 49/613. (line cycle) 83/85 96/97 95/984. (sunscreen) 65/76 75/83 80/905. (fruits & veggies) 58/65 50/62 56/656. (bicycle on hill) 45/52 54/60 50/587. (floating block) 30/34 25/31 30/368. (inference bag) 34/43 43/58 59/679. (seed experiment) 29/36 43/61 49/62

Constructed Response1. (river in flood plain-a) 48/62 66/83 66/832. (river in flood plain-b) 16/23 21/28 28/453. (snow on mountains) 31/46 36/53 61/734. (animal protection-a) 46/60 61/77 87/925. (animal protection-b) 29/42 48/64 48/646. (candle in jar) 49/64 52/62 80/857. (sugar masses) 27/37 28/43 61/758. (air pollution-a) 31/48 45/59 52/679. (air pollution-b) 21/34 37/48 35/51

10. (oil spills) 16/27 27/46 51/64


The science supervisor’s rating of the school-district teachers using the rubric described in

Table 2 constituted yet another threat to our study’s validity. The supervisor was given free reign in

assigning the ratings. For practical reasons of an overcrowded schedule and a reluctance to

jeopardize trust between the supervisor and her staff, we could not impose a strict set of guidelines

in applying the rating rubric. Thus, some ratings were based on extensive interaction and ob-

servation data while other ratings were based on very limited data. A more rigid set of rating

guidelines and a demand for more exhaustive evidence on which to base ratings may well have

produced a more variable and different set of quality of implementation ratings.

It may be questionable to believe that inquiry science teaching of any kind can uniformly

stimulate the desired achievement gains in elementary school students with any or all generalist

teachers. Inquiry science teaching may place too great of conceptual, pedagogical, logistical, and

resource demands on generalist teachers in the current elementary school environment. If inquiry

science teaching is an ineffective instructional approach in this context, one would not expect to

detect the associated gains in conceptual and affective growth of all students involved. The 1980

meta-analyses and the research base for the science education reform in the United States did not

suggest such an interpretation (American Association for the Advancement of Science, 1993;

Shymansky, Kyle, & Alport, 2003). We continue to believe that the interactive-constructivist

approach (modified learning cycle) to inquiry teaching with cross-curricular connections to

language arts, mathematics, and social studies has unrealized potential for self-contained

classrooms involving generalist teachers.

Teacher workshops focused on inquiry modules specific to their teaching responsibilities and

developing associated instructional resources to adapt the science kits to their classroom and

students may not be an effective professional-development strategy. These instructional activities

do not fully reflect the complete challenge for teachers of implementing a new instructional

resource into their teaching that involves changing pedagogical-content knowledge, beliefs about

science and science teaching, and classroom practices, and opening the implicit beliefs of their

students to such innovative curriculum and instruction. This alternative interpretation may well

have merit, but the NSF with its call for proposals for the reported research projects appears to have

disregarded the likelihood of such an interpretation. The professional-development activities of

Science PALs were designed around the authentic work that elementary teachers do outside of

their classrooms in preparing to teach a specific science topic and providing the ongoing support

of a science advocate in their school and a science partner in their school district to help with

the actual classroom implementation. We believe the publication of research reports such as this

may influence funding agencies to have a much more open mind to what constitutes effective

professional development and to recognize the complexities and difficulties of systemic change.

Finally, perhaps no significant student learning effects were found because the Science PALs

program itself is not a robust and intense approach and the research design was not sensitive to the

changes that did occur. We considered these interpretations at the outset of the research design and

during the data analysis and interpretation. Qualitative responses reported elsewhere indicate that

teachers believe and support the parental involvement, school–home activities, and literacy

approach to teaching elementary science (Shymansky et al., 2000; Yore et al., in press). There are

convincing data that Science PALs itself works in some circumstances and that in this instance, its

failure to have a significant impact may be related to the difficulties teachers have enacting their

newly developed understanding into classroom practice, the lack of sensitivity of the measures

of science understanding and attitudes, or the quasi-experimental approach did not capture the

positive effects in some situations. But, like many controlled implementations of teaching stra-

tegies and instructional materials in selective contexts and participant samples that demonstrate

positive effects, similar effects are not generally found when a larger clientele pool is involved in


the implementation. This may be the case in the 16 elementary schools of the host school district.

Exemplar cases of selected data from some schools, teachers, and students could tell a much more

exciting story, and such a case study of successful subsystems within the larger system might well

provide insights for future research and teacher enhancement projects. Unfortunately, this study

was about systemic change involving all participating elementary schools and classrooms.

Conclusion

The results of our study support the Supovitz and Turner (2000) claim that teacher effects of

professional development should not be expected until teachers have experienced a minimum of

about 80 hr of work and suggest that student effects should not be expected for at least 3 or more

years after that. Our results also reinforce the conclusions of Loucks-Horsley and Matsumoto

(1999), who pointed out how difficult it is to study, let alone clearly establish, direct cause–effect

connections between professional development and student performance. Yet, we have no choice

as professional educators but to continue to study those connections, lest we admit that we are only

‘‘stabbing in the dark’’ with our professional-development promises and practices. Without solid

evidence that our professional-development activities do result in enhanced student performance,

we can only claim ‘‘artistic license’’ for what we do in teacher education. Despite the difficulties

and cost of studying the connection between professional development and student performance,

we have to persevere.

Notes

This article is based upon research supported by National Science Foundation Grant ESI-9911857.

Any opinions, findings, and conclusions or recommendations expressed in this article are those of the

authors and do not necessarily reflect the views of the National Science Foundation.

References

American Association for the Advancement of Science. (1990). Science for all Americans.

New York: Oxford University Press.

American Association for the Advancement of Science. (1993). Benchmarks for science

literacy. New York: Oxford University Press.

American Association for the Advancement of Science. (2001). Atlas of science literacy.

Washington, DC: Project 2061.

Darling-Hammond, L. (1996). What matters most: Teaching for America’s future (Summary

Report). New York: National Commission on Teaching and America’s Future.

Dunkhase, J., Hand, B., Shymansky, J.A., & Yore, L.D. (1997, November). The effects of a

teacher enhancement project designed to promote interactive-constructive teaching strategies in

elementary school science on students’ perceptions and attitudes. Paper presented at the Annual

School Science and Mathematics Association Conference, Milwaukee, WI.

Education Commission of the States. (1997). Investing in teacher professional development:

A look at 16 school districts. Denver, CO: Author.

Ford, C., Yore, L.D., & Anthony, R. (1997, March). Reforms, visions, and standards: A cross-

cultural curricular view from an elementary school perspective. Paper presented at the annual

meeting of the National Association for Research in Science Teaching, Oak Brook, IL.

Frechtling, J., Sharp, L., Carey, N., & Vaden-Kiernan, N. (1995, June). Teacher enhancement

programs: A Perspective on the last four decades. Washington, DC: National Science Foundation.


Garet, M., Porter, A., Desimone, L., Birman, B., & Yoon, K. (2001). What makes professional

development effective? Results from a national sample of teachers. American Educational

Research Journal, 38, 915–945.

Guskey, T. & Sparks, D. (1996). Exploring the relationship between staff development and

improvements in student learning. Journal of Staff Development, 17, 34–38.

Impara, J.C. (2001, April). Alignment: One element of an assessment’s instructional utility.

Paper presented at the National Council on Measurement in Education, Seattle, WA.

International Association for the Evaluation of Educational Achievement. (1997). TIMMS

science items (third and fourth grades). Chestnut Hill, MA: Author.

Lavoie, D.R. & Roth, W.-M. (Eds.). (2001). Models of science preparation: Theory into

practice. Dordrecht, The Netherlands: Kluwer.

Loucks-Horsley, S. & Matsumoto, C. (1999). Research on professional development for

teachers of mathematics and science: The state of the scene. School Science and Mathematics, 99,

258–271.

Messick, S. (1989). Validity. In R.L. Lenn (Ed.), Education measurement (3rd ed., pp. 13–

103). New York: Macmillan.

National Commission on Teaching and America’s Future. (1996). What matters most:

Teaching for America’s future. New York: Author.

National Research Council. (1996). National science education standards. Washington, DC:

National Academy Press.

Norris, S.P. & Phillips, L.M. (2003). How literacy in its fundamental sense is central to

scientific literacy. Science Education, 87, 224–240.

Shields, P.M., Marsh, J.A., & Adelman, N.E. (1998). Evaluation of NSF’s Statewide

Systemic Initiatives (SSI) Program: The SSI’s impacts on classroom practice. Menlo Park, CA:

SRI.

Shymansky, J.A., Kyle, W.C., Jr., & Alport, J.F. (2003). The effects of new science curricula

on student performance. Journal of Research in Science Teaching, 40, S68–S85.

Shymansky, J.A., Woodworth, G., Norman, O., Dunkhase, J., Matthews, C., & Liu, C.T.

(1993). A study of changes in middle school teachers’ understanding of selected ideas in science as

a function of an inservice program focusing on student preconceptions. Journal of Research in

Science Teaching, 30, 737–755.

Shymansky, J.A., Yore, L.D., Dunkhase J.A., & Hand, B.M. (1998, January). Science,

parents, activities, and literature: Overview, results, and reflections. Paper presented at the Inter-

national Conference of the Association for the Education of Teachers in Science, Minneapolis,

MN.

Shymansky, J.A., Yore, L.D., & Hand, B.M. (2000). Empowering families in hands-on

science programs. School Science and Mathematics, 100, 48–56.

Shymansky, J.A., Yore, L.D., Treagust, D.F., Thiele, R.B., Harrison, A., Waldrip, B.G.,

Stocklmayer, S.M., & Venville, G. (1997). Examining the construction process: A study of

changes in level 10 students’ understanding of classical mechanics. Journal of Research in Science

Teaching, 34, 571–593.

Slavin, R.E. (2003). A reader’s guide to scientifically based research. Educational

Leadership, 60(5), 12–16.

Smith, D.C. (1999). Changing our teaching: The role of pedagogical content knowledge in

elementary science. In J. Gess-Newsom & N.G. Lederman (Eds.), Examining pedagogical content

knowledge (pp. 163–197). Dordrecht, The Netherlands: Kluwer.

Supovitz, J.A. & Turner, H.M. (2000). The effects of professional development on science

teaching practices and classroom culture. Journal of Research in Science Teaching, 37, 963–980.


Webb, N.L. (1997). Criteria for alignment of expectations and assessments in mathematics

and science education [Research Monograph No. 6]. Washington, DC: Council of Chief State

School Officers.

Webb, N.L. (1999). Alignment of science and mathematics standards and assessments in four

states [Research Monograph No. 18]. Madison, WI: National Institute for Science Education.

Weiss, I.R., Montgomery, D.L., Ridgway, C.J., & Bond, S.L. (1998). Local systemic change

through teacher enhancement: Year Three Cross-Site Report. Chapel Hill, NC: Horizon Research.

Yore, L.D., Shymansky, J.A., & Anderson, J.O. (in press). Sensing the impact of elementary

school science reform: A study of stakeholder perceptions of implementation, constructivist

strategies and school–home collaboration. Journal of Science Teacher Education.

Yore, L.D., Shymansky, J.A., Henriques, L., Hand, B.M., Dunkhase, J.A., & Lewis, J.O.

(1998, January). Students’ perceptions of science teaching and attitudes toward science learning

and teachers’ self-report of using children’s ideas, applications of science, and use of print

resources as indicators of classroom teaching. Paper presented at the International Conference of

the Association for the Education of Teachers in Science, Minneapolis, MN.

Zembal, C., Starr, M., & Krajcik, J. (1999). Constructing a framework for elementary science

teaching using pedagogical content knowledge. In J. Gess-Newsom & N.G. Lederman (Eds.),

Examining pedagogical content knowledge (pp. 237–256). Dordrecht, The Netherlands: Kluwer.


impact of a school district's science reform effort on the achievement and attitudes of third-...

Documents