Evaluating Opportunity to Learn — page 3 Fidelity of Implementation Project — page 6 FOI and the Secondary Component — page 9 CSMC Update — page 10 Contact Information — page 11

continued on next page

UCSMP Turns 25Share a UCSMP Story by January 11

In 2008, UCSMP will turn 25! You are invited to join the anniversary celebration by sharing a “UCSMP Story” that you think would interest others.

UCSMP ideas, materials, international conferences, translations, users conferences, and institutes may have influenced your thinking, or the thinking of others in your district. We welcome essays on the Project’s impact and classroom stories, from vignettes to longer pieces.

Please submit your story, with your name and contact information, by Friday, January 11, 2008. Stories can be emailed to ucsmp@uchicago.edu (please use “UCSMP Story” as the subject line), or mailed to:

UCSMP Newsletter UCSMP 6030 S. Ellis Avenue Chicago, IL 60640.

Selected stories will be considered for publication in a Newsletter or, possibly, a monograph. They may be edited for length, clarity, or to fit our editorial guidelines. Authors will receive edited versions for their approval before publication.

The University of Chicago School Mathematics Project

No. 38 Fall 2007–08

UCSMP Newsletter

Inside this issue . . .

Two Early Users of the Third Edition of Everyday Mathematics

During the 2006–2007 school year, schools across the country began using the Third Edition of Everyday Mathematics. This is a report on the district-wide

use of Everyday Mathematics in the Virginia Beach City Public Schools and the Fairbanks North Star Borough School District.

Virginia Beach: Successful Upgrade from the Second Edition

With a population of nearly 500,000, Virginia Beach is the most populous city in Virginia. It is one of the Seven Cities of Hampton Roads, which also include

Chesapeake, Norfolk, and Newport News. Best known as a

beach resort, Virginia Beach is also home to several state parks, three military bases, and a number of large corporations.

The Virginia Beach City Public Schools serve a suburban district that has approximately 71,000 students in 57 elementary schools, 15 middle schools, and 11 high schools. Approximately 26% of the diverse student population qualifies for free or reduced-price meals.

After having used both the First and Second Editions of Everyday Mathematics, the Virginia Beach City Public Schools opted for a system-wide adoption of the Third Edition for the 2006–2007 school year. Teachers and administrators in the district had high hopes for the Third Edition, including more defined assessment options, differentiation options, more content background included in the Teacher’s Lesson Guides, a reference book for first and second grades, an improved kindergarten program, and more writing opportunities.

Were teachers and administrators satisfied with the changes made in the Third Edition? Absolutely. “I love it!” Judy Fisher, Elementary Mathematics Coordinator for the district, says of the Third Edition. Fisher, who supports mathematics for Grades K–5, has worked with all three editions of Everyday Mathematics and reports that the Third Edition is an improvement on the previous editions. Among the many highlights that Fisher lists are My Reference Book (the new reference volume for first and second grades), revisions to the Kindergarten program, and the addition of the Differentiation Handbook. She is also pleased by the way Part 3 of each lesson offers options that relate directly to the Key Concepts and Skills taught in Part 1 of the lesson.

Kathy Sands, a kindergarten teacher in Virginia Beach and consultant for Everyday Mathematics, is just as pleased as Fisher. “It is all that I could have dreamed up and more,” says Sands. She felt that there was not enough emphasis on number writing, graphing, and problem-solving in the First and Second Editions of Kindergarten Everyday Mathematics. With

UCSMP Newsletter

Fall 2007–08 Page 2

Third Edition EM Users, continued from previous page

NEW EM MATERIALS IN SCHOOLS

the addition of My First Math Book and the Survey Routine, Sands says that these are no longer concerns.

Fairbanks, Alaska: Successful All-School Introduction of Everyday Mathematics Third Edition

Over 4,000 miles away, in Fairbanks, Alaska, educators were striving for the same success with the Third Edi-tion of Everyday Mathematics. Fairbanks is the largest

city in the Interior region of Alaska, and the second largest city in the state. With a population of over 30,000, Fairbanks is contained within the North Star Borough, similar to a county, but encompassing an area of over 7,000 square miles, greater than the area of the entire state of Connecticut!

The Fairbanks North Star Borough School District serves nearly 15,000 students in a total of 33 different schools, including 19 elementary schools and schools on two military installations. Six of the schools are Title I schools and one receives Targeted Assistance.

While individual schools have been using the Second Edition of Everyday Mathematics for several years, the 2006–2007 adoption of the Third Edition was the first all-district mathematics adoption. Peggy Carlson, Curriculum Coordinator for the district, recalls, “We had a challenging year of implementation. It was worth all the effort we put in, as we saw an increase in our standardized test scores.” While Carlson insists that the test scores are not the final goal, she admits that higher test scores help assuage any doubts held by parents and school board members.

Carlson reports that teachers who taught the Second Edition found the Third Edition more teacher-friendly and easier to navigate than the Second Edition. Those who had not previously taught the program had a wide range of reactions.

Lindy Kinn, Math Support for Grades 4–6, was a new user of the program. “The program incorporates all the best practices I’ve learned over the years,” reports Kinn. She states that the program was met with mixed reaction by the parents new to the program. Parents of students who historically struggle with mathematics felt that the program was too challenging. Once teachers began using Readiness activities to preview content prior to teaching lessons, parents’ concerns diminished. On the other end of the spectrum, parents of high-achieving students were surprised that their children did not always grasp concepts immediately. These parents had to be reassured that challenging their children was a positive outcome of Everyday Mathematics.

According to Kinn, “There are still some teachers who

were not comfortable with the spiral and can’t get used to not teaching to mastery the first time. Not getting a concept right away was hard for teachers, parents, and students. The change in pedagogy is not an easy shift.” By the end of the year, however, despite struggles, Carlson and Kinn report that most teachers saw the value in Everyday Mathematics. And, as teachers became more comfortable with the program and the spiral, attitudes of both students and parents relaxed.

Third Edition Differentiation and Assessment Meet with Approval

Virginia Beach teacher Kathy Sands cites the emphasis on differentiation in the Third Edition as one of its greatest strengths. Everyday Mathematics offers a

wide variety of options that give all students an opportunity to engage with the lesson content.

Lindy Kinn, in Fairbanks, echoes this sentiment. For Kinn, the differentiation in the program is the most salient feature, as it allows teachers to teach each student at his or her level. She says that even Gifted and Talented students are challenged.

Both Judy Fisher and Kathy Sands, in Virginia Beach, name the revamped assessment program as one of the most outstanding features of the Third Edition. There is a “greater emphasis on assessment and giving teachers those ‘teachable moments’ to assess in a natural way,” reports Sands. Fisher adds that changes to the Progress Check lessons, including the addition of an Open Response assessment and the division of the Written Assessment into two parts, one suitable for grading and one for planning future instruction, are an improvement on the Second Edition. Joan Harwin, a Virginia Beach fifth-grade teacher, adds that the new self-assessments included in the Third Edition are extremely beneficial to both students and teachers. They give “tremendous insight into how students feel about what they have studied.” The new organization of goals into Program Goals, Grade-Level Goals, and Key Concepts and Skills rounds out the highlights of the assessment program.

Hopes for Continued Success

The students in Virginia Beach have made a smooth transition to the Third Edition. According to Fisher, Everyday Mathematics teaches in the way that children

learn. Furthermore, as a result of frequent assessment, chil-dren know how they are doing and, if they are not “making adequate progress,” additional opportunities are available to help them find success.

In Fairbanks, Everyday Mathematics has been successfully adopted in a far-flung district in which many teachers were first-time users. In the end, administrators, teachers, students,

UCSMP Newsletter

Fall 2007–08 Page 3

Opportunity to Learn: A Critical Variable in UCSMP Curriculum Research

Throughout its history, UCSMP has field-tested its secondary level curriculum materials prior to their commercial publication to obtain feedback from

teachers and students about the materials. One of the most persistent findings of our research and of research by others on the use of curriculum materials is that students’ opportunity to learn mathematics varies based on curricula. This variation occurs among teachers using the same curriculum materials as well as between teachers using different curriculum materials. In this brief essay, we describe some of these differences and their implications for UCSMP research.

Variations in Opportunity to Learn among Classrooms Using the Same Curriculum

In our field tests of UCSMP materials, teachers regularly complete Chapter Evaluation forms in which they indicate which lessons they taught, the number of days the lesson

took, and the problems assigned; this information provides some indication of the extent to which teachers implemented the curriculum. We have found that teachers follow many different paths through their textbook. Some start with Lesson 1-1 and proceed in order through the book. Others skip one or two sections in each chapter. So, by the end of the year, even teachers who have taught the same number of lessons have often taught different topics.

For instance, Table 1 (right) reports results from the seven teachers using the field-trial version of the Third Edition of Transition Mathematics. Teachers A, E, and F taught almost all lessons in Chapters 1–8, but covered various amounts of Chapters 9–12. In contrast, Teacher B regularly skipped lessons from each of

Chapters 1–8, but covered more of Chapters 9–12 than the average teacher in the study. (Typically the lessons skipped emphasized content that was not part of his state’s curricular requirements for Grade 7.) Teachers C1, C2, and D taught most of the lessons in the first third of the book and the majority of lessons in the second third but none of the lessons in the final third of the book. Thus, even though the overall percent of lessons taught for two teachers may be fairly close, for example, Teachers B (73%) and D (67%), and even though the teachers were using the same curricular materials, students of these teachers had very different opportunities to learn mathematics. Similar variability in opportunity to learn is also evident in the percent of problems assigned to students reported by each teacher.

In addition to the Chapter Evaluation form, on every posttest that UCSMP administers, we ask all teachers participating in the study to answer the following opportunity-to-learn (OTL) question for each item on the posttest: “During this school year, did you teach or review the mathematics needed for your students to answer this item correctly?” Teachers respond in one of four ways: (1) Yes, it is part of the text I used; (2) Yes,

continued on next page

Ch 1–4 (n = 44)

Ch 5–8 (n = 55)

Ch 9–12 (n = 42)

Overall (n = 141)

Ch 1–4 (n = 903)

Ch 5–8 (n = 1028)

Ch 9–12 (n = 706)

Overall (n = 2637)

A 98 95 33 80 98 95 33 79B 82 75 62 73 73 65 47 63C1 98 89 0 65 79 84 0 60C2 93 62 0 56 93 62 0 56D 100 91 0 67 73 70 0 52E 95 100 40 81 71 73 28 60F 100 98 67 89 98 93 55 85

Overall 95 87 29 73 84 78 23 65

Table 1. Opportunity to Learn the Third Edition of Transition Mathematics by Teacher

Teacher Percent of Lessons Taught Percent of Problems Assigned

Note: Teachers C1 and C2 are in the same school.

OPPORTUNITY TO LEARN

and parents viewed the change positively. The challenge in future years is to assist teachers in becoming more comfortable

with Everyday Mathematics. As the teachers become more accepting, so too will parents and students.

UCSMP Newsletter

Fall 2007–08 Page �

in the study, UCSMP and comparison classes. Comparison teachers also complete the OTL form previously discussed. Whether teachers are responding to an OTL measure for a standardized test or a test constructed by UCSMP, we have found that there is considerable variability across teachers and schools in the number of items for which teachers report Yes on the OTL form. This is true whether one looks only at the items common to the teachers in the same school teaching a given course with the UCSMP or comparison curriculum or one looks at the items common to all teachers participating in the study across multiple schools and multiple curricula. The variability is robust, in that the same patterns of variability are found for middle school curricula, such as Transition Mathematics, or for high school curricula, such as Advanced Algebra.

Table 2 (next page) reports the number and percent of items common to teachers at the same school and across schools for evaluation studies of four different courses conducted over more than a decade; note that in most cases the comparison teachers were using a non-UCSMP curriculum. As the results indicate, at the school level, the number of items common to both teachers at the school might be as low as 47% (in the Advanced Algebra study) or as high as 100% (in the Transition Mathematics study). Across all the teachers in a given study, the number of common items is much lower, ranging between 34 to 48% for the four tests highlighted in Table 2. In general, the set of items that all teachers in a particular study report having taught assess Skills, but not Properties, Uses, or Representations of the topics taught in the course.

Reporting Achievement Data to Control for Opportunity to Learn

Many individuals are surprised by the variability that exists among classes using the same curriculum and between classes using different curricula, even

in a relatively small number of schools. Given differences in

although it is not part of the text I used; (3) No, because it is not part of the text I used; or (4) No, although it is part of the text I used.

Figure 1 (below) illustrates responses to this question from the six teachers using the field-trial version of UCSMP Alge-bra (Third Edition) to the 32 items on the Terra Nova Algebra Test, a standardized test used during 2005–2006 as part of the evaluation study of UCSMP Algebra. Each column represents one of the items on the test. Each row represents one teacher’s responses to the OTL question for each item. Gray indicates the teacher reported answering “yes” to the OTL question; white indicates that the teacher answered “no.” The figure again shows that even when teachers are using the same cur-riculum, there is considerable variability in opportunities to learn mathematics. Of the 32 items on the test, just 50% were reported as taught by all six teachers participating in the study. However, 30 of the 32 items were reported taught by at least four of the six teachers, indicating that the test was reasonable over the content of the textbook.

Variations in Opportunity to Learn between Classrooms Using Different Curricula

Given the variability that exists among classes using the same curriculum, as illustrated in Table 1 and Figure 1, it is natural to wonder about the variability in op-

portunity to learn between classes using different curricula. This issue is of interest to UCSMP because our research design typically involves matched pairs of classes in schools, with one class using the UCSMP field-trial curriculum and the other class using whatever curriculum was already in place at the school (sometimes an earlier edition of the UCSMP text and sometimes a non-UCSMP comparison curriculum).

UCSMP administers posttests to all classes participating

Figure 1. UCSMP Algebra (Third Edition) teachers’ responses to the OTL item for the Terra Nova Algebra Test. (Shadow indicates the teacher reported teaching or reviewing the necessary content.)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ABCDEF

OTL, continued from previous page

OPPORTUNITY TO LEARN

UCSMP Newsletter

Fall 2007–08 Page �

opportunity to learn for students as documented in Tables 1 and 2 and Figure 1, how can achievement be reported in a responsible way to highlight what students are able to do? After all, if students have not had an opportunity to learn a particular topic, then low achievement may not be unexpected, and should not necessarily be blamed on the student or the curriculum.

UCSMP has addressed this difficulty by reporting achieve-ment in three different ways. In each case, we report results at the class level, rather than aggregating the data into a UCSMP and non-UCSMP group. First, we report overall achievement on the Entire Test and report the percentage of items for which the teacher responded Yes on the OTL questionnaire; in this way, readers of the data can look at any differences and determine the extent to which they believe results might be influenced by OTL. Second, we analyze the achievement results at the school level using only those items for which both teachers at the school indicated that students had a chance to learn the content needed to answer the items; this set of items, which we call the Fair Test, controls for opportunity to learn at the school level. Third, we analyze the data using only those items for which all teachers in the study reported Yes on the opportunity to learn measure. We call this set of items the Conservative Test. In each study, we report the mean percent correct on the Entire Test, Fair Test, and Conservative Test. A repeated-measures t-test is used to compare the differences in the means for the pairs of classes to determine whether an overall difference in achievement based on cur-ricula exists for each of the three ways of reporting achievement.

In studies of the Second Edition of UCSMP Advanced Algebra, significant differences in achievement between students using the UCSMP curriculum or the comparison curriculum were found on the Entire Test and the Fair Tests, but not on the Conservative Test. However, the Conservative Test consisted of only 15 of the 36 items on the test, most of which were skills. At present, content-specific standardized tests do not seem to exist beyond first-year algebra so UCSMP has no choice but to develop its own test. Since the Fair Tests use only items at each school for which teachers reported teaching the content, the results should not be considered biased toward UCSMP by focusing only on items in the UCSMP curriculum. As the Conservative Test consists of only those items for which

all teachers reported teaching the content, this measure consists of items that might be perceived as a core set of items for second-year algebra, regardless of curriculum used. However, this measure is also highly controlled by the teacher who covers the least amount of content.

Similar results have been found in other UCSMP studies. In studies of the Second Edition of UCSMP Geometry, no significant differences in achievement between students using the UCSMP curriculum or the comparison curriculum were found on the standardized measure, regardless of which of these three methods of reporting were used. In contrast, on the UCSMP-constructed test, significant differences were found on the Entire Test and the Fair Tests; the Conservative Test could not be utilized because there were only four of 35 items for which all teachers reported teaching the content.

Conclusion

Our attention to opportunity to learn as an important variable in curriculum research highlights similari-ties and differences among classes using the same

UCSMP curriculum as well as among classes using the UCSMP curriculum and its comparison curriculum. Report-

continued on next page

Course Being Studied

Transition Mathematics Algebra Geometry Advanced

AlgebraYear of data collection 05–06 05–06 93–94 93–94

Type of test Written by UCSMP

Terra Nova Algebra,

standardized

High School Subjects Test:

Geometry, standardized

Written by UCSMP

Total number of items on test 40 32 40 36

Number of items common

to both UCSMP and comparison

teachers in the same school

Ranges from 27 to 40

(68%–100%)

Ranges from 19 to 31

(59%–97%)

Ranges from 26 to 32

(65%–80%)

Ranges from 17 to 26

(47%–72%)

Number of items common to all teachers in the study

across schools

16 (40%)

11 (34%)

19 (48%)

15 (42%)

Table 2. Variability in Opportunity to Learn by Course and Test

OPPORTUNITY TO LEARN

UCSMP Newsletter

Fall 2007–08 Page �

ing achievement in three ways that control for opportunity to learn at the teacher, school, or study level enables us to report differences while being as objective as possible. For critics who believe that a test constructed by UCSMP is inherently biased, the results of the OTL questionnaire suggest that a properly-constructed Project test can be just as reasonable as a standardized test. Tests are always reflections of our value system in terms of what we view as important. By using OTL as a measure to control for differences in values, we believe we provide a robust and honest perspective on UCSMP achieve-ment. If anything, our attention to opportunity to learn has kept us from attempting to assess many ideas that we know are taught only in the UCSMP curriculum.

Looking at OTL also provides a perspective on the degree to which a given UCSMP curriculum is implemented as ex-pected. The degree of implementation helps explain differences in achievement across multiple schools, even when the same curriculum is used.

For detailed information about the use of OTL, please consult the technical reports of the evaluation studies for the second editions of Transition Mathematics, UCSMP Algebra, UCSMP Geometry, and UCSMP Advanced Algebra (available from the UCSMP general office) or the references listed below.

References

Thompson, D. R., & Senk, S. L. (2001). The effects of curriculum on achievement in second-year algebra: the example of the University of Chicago School Mathematics Project. Journal for Research in Mathematics Education, 32, 58–84.

Thompson, D. R., & Senk, S. L. (2006). Methods for controlling for opportunity-to-learn. In S. Alatorre, J. L. Cortina, M. Sáiz, A. Méndez (Eds.), Proceedings of the twenty-eighth annual meeting of the North American chapter of the International Group for the Psychology of Mathematics Education, (pp. 179–186, Volume 2). Mérida, Mexico: Universidad Pedagógica Nacional..

Overview

The University of Chicago’s Center for Elementary Mathematics and Science Education (CEMSE) has received nearly one million dollars in funding from the

National Science Foundation to study the fidelity of implemen-tation of science and mathematics instructional materials in the Chicago Public Schools. The project, titled, “Applied Research on Science Materials Implementation: Bringing Measurement of Fidelity of Implementation (FOI) to Scale” began in January 2007 and will continue through December 2009.

The first goal of the project is the development of a suite of instruments to measure the implementation of standards-based science and mathematics instructional materials at the K–8 level. The second goal is to use the instruments to measure the fidel-ity of materials implementation in the Chicago Public Schools. The third goal is the exploration of relationships between types of implementation and student outcomes. To accomplish these goals, the project is producing instruments for measuring implementation of five specific curricula: Everyday Mathemat-ics and the science programs FOSS, STC, Science Companion, and SEPUP. The project will also develop a User’s Guide that

Fidelity of Implementation: What Is It and How Do You Measure It?

CEMSE’s NSF-Funded Project in the Chicago Public Schools

describes procedures for using the instruments and adapting them for use with other instructional materials. With the Guide, a range of school-based and district-based practitioners will be able to use the instruments to better understand FOI of their programs.

The Fidelity of Implementation project hopes to further mathematics and science education in several ways. It is supporting the growing call for high-quality research in education by developing and disseminating instruments to study the efficacy and effectiveness of instructional materials programs. Through rigorous definitions and measurements, it is attempting to shed light on relationships between different types of implementation and student outcomes. The project is also hoping to clarify what “fidelity of implementation” of instructional materials means.

What is “Fidelity of Implementation”?

Although there is some shared understanding in the field of what FOI is, there is no agreement about how to describe or define it at any level of detail. Before

the project could proceed, the CEMSE team had to decide on a definition. That definition is: the extent to which an enacted

OTL, continued from previous page

OPPORTUNITY TO LEARN

UCSMP Newsletter

Fall 2007–08 Page 7

focus on expectations for what the teacher needs to do. The educative critical components, on the other hand, represent the expectations for what the teacher needs to know.

In contrast, Instructional Critical Components are those elements of the program that reflect the developers’ beliefs about appropriate pedagogy and classroom interactions. These components are also divided into sub-categories. Pedagogical critical components reflect the developers’ expectations about 1) the instructional strategies the teacher needs to employ and 2) the interactions the teacher needs to orchestrate in order to use the program appropriately. Similarly, the student engage-ment critical components reflect the developers’ expectations for student behaviors and interactions during instruction.

Implementation Types: By separating structural and instruction-al critical components, the CEMSE team will be able to analyze relationships between them. For example, the team will be able to answer questions such as, “Do the teachers who read the educative content background information demonstrate more thoughtful questioning and discussions in their classrooms?” or, “Is there a relationship between teacher use of optional activities and student participation in small groups?”

Further, the critical component organization will allow the CEMSE team to identify what they refer to as types of imple-mentation. The team will identify three to seven types and then explore relationships between those types and desired student outcomes. In other words, they will seek to answer the questions, “When teachers use Everyday Mathematics in a particular way, does that type of use lead to improved student outcomes? Why or why not?”

A Note About Content: Content is essential to any instructional materials program. At first, the FOI team considered the pos-sibility that “content” would itself be a critical component category. Then, on further consideration, the team determined that “content” is not a category of its own, but rather, a critical component that resides in both the structural and the instruc-tional categories. On the structural side, the extent to which the enacted instruction is consistent with the intended content is a critical component. On the instructional side, the extent to which the content presented in the classroom is appropriately matched and balanced in breadth and depth (balancing facts, procedures, processes, and concepts) for the students and the nature of the particular lesson is a critical component.

Level of Differentiation: The framework also indicates which critical components are common across programs and which are unique. The ability to rigorously measure the FOI of the critical components, and to understand which characteristics are and are not shared is a basic goal of the FOI project and an

Procedural Educative Pedagogical Student Engagement

Common to Math

and Science

MaterialsCommon to Science Materials

OnlyCommon to Math

Materials Only

Program Specific

FOI for Instructional Materials

FOI Framework with Sample Critical Components

Structural Critical Components

Instructional Critical Components

program is consistent with the intended program. In order to operationalize this definition so that the team could begin to work in schools, they had to establish the “intended program” for each of the curricula. They determined that programs are made up of explicit and implicit essential elements or “critical components.” In other words, “critical components” are the elements of the program that make it, “it.” Thus, the operational definition of FOI is: the extent to which the critical components of an intended program are present when that program is enacted.

Organizing Critical Components

Measurement of critical components is at the core of mea-suring FOI. Therefore, CEMSE organized the critical components into categories that would facilitate their

measurement and later support a more meaningful analysis of types of implementation and their relationships to student outcomes.

Here is the framework:

The framework has two main categories of critical compo-nents, structural and instructional.

Structural Critical Components are those elements of the written instructional materials that reflect the developers’ be-liefs about appropriate, effective materials design. The “Struc-tural” components are further classified into sub-categories. Procedural critical components indicate the expected step-by-step actions a teacher is expected to take. In other words, they

continued on next page

FIDELITY OF IMPLEMENTATION

UCSMP Newsletter

Fall 2007–08 Page 8

essential part of any experimental or quasi-experimental compari-son study of these materials. Without identifying shared critical components, researchers cannot sufficiently distinguish programs in order to determine which performs better than another.

Identifying Critical Components of Everyday Mathematics

With the FOI framework in place, CEMSE began the pro-cess of identifying the critical components in Everyday Mathematics and in each science instructional materi-

als program. Some components are unique to a single program; others are shared across one or more. The process began with a thorough review of the materials by a CEMSE FOI Project team member. It evolved to include conversations with the developers of the programs and with teachers who use the program.

As the team identified critical components, it became clear that many of them—particularly those on the “Instructional” side of the framework—are shared across mathematics and science pro-grams. For example, in the “Instructional-Pedagogical” column, Everyday Mathematics shared a number of critical components with other programs including: “Teacher use of Materials, Ma-nipulatives and Tools,” “Teacher Facilitation of Student Discus-sion,” “Teacher Facilitation of Group Work,” “Teacher Building on/Stimulating Student Interest,” and “Teacher use of a Variety of Instructional Formats and Organizations.” All of these are elements of the instruction that one would expect to see in an Everyday Mathematics classroom (as well as classrooms that teach other reform-oriented mathematics and science programs).

Similarly, Everyday Mathematics has critical components in the Instructional-Student Engagement column that it shares with other programs. For example, in an Everyday Mathematics classroom one would expect to see: “Students converse with one another,” “Students draw reasoned conclusions,” and “Students contribute to small group work.” If students aren’t engaged in these behaviors, one might call into question whether Everyday Mathematics is be-ing implemented with fidelity—in other words, whether it is being taught in a manner that is true to the intentions of the developers.

The “Structural” side of the framework also houses shared critical components. For example, all of the reform-based programs had a particular “Investigation or Lesson Order,” a “Lesson Overview,” and called for “Inclusion of all Essential Activities” within a lesson. The CEMSE team has given these critical components common names, but when developing the instruments, will customize each instrument to the program. So, for example, this last critical component, “Inclusion of all Es-sential Activities,” for Everyday Mathematics includes: “Mental Math and Reflexes,” “Math Message,” “Teaching the Lesson,” “Ongoing Learning and Practice,” and “Games.”

Finally, the Structural-Educative column of the framework also houses shared critical components such as “ Lesson-level background information on content,” “Unit-level background information on pedagogy,” and “Information on Standards and Benchmarks.”

Critical Components Unique to Everyday Mathematics: Even though many critical components were common across programs, a number of critical components were unique to their programs. Most of these fell into the structural-procedural column. For Everyday Mathematics, unique structural-procedural critical components included “Student Reference Books” and “Slates.”

Limitations of the Project

The FOI project focuses squarely on describing instruction in the classroom. However, the team recognizes that understanding the contexts and

conditions that influence implementation (ranging from teacher characteristics to administrative support to pro-fessional development) is also an essential part of school improvement. The FOI instruments gather information on these contexts and conditions in a limited way, when the additional data collection does not interfere with the primary goal—understanding FOI.

Status of the Project and Next Steps

The CEMSE team is just completing drafts of all of the instruments and is working with the Chicago Public Schools to conduct a pilot test in fall 2007. Then,

CEMSE will revise the instruments based on the pilot test and conduct a field test in spring 2008. The third year of the project will include refinement of the instruments, preliminary data analysis, and completion of the User’s Guide. The instru-ments include teacher and principal questionnaires, classroom observation protocols, school walk-through protocols, logs, and teacher, principal, and science specialist interview pro-tocols. CEMSE expects and hopes that these instruments and the User’s Guide will be of use not only to researchers, but also to school and district leaders.

Project progress and findings will be shared in an on-going manner through CEMSE’s web site at http://cemse.uchicago.edu . This site will house the project’s papers and presentations, technical reports, and the instru-ments and User’s Guide. It will also house a forum for others studying FOI. If you have any questions about the FOI project or want more information on the critical components of Every-day Mathematics, please contact Jeanne Century, CEMSE’s director of science education, at jcentury@uchicago.edu .

FOI, continued from previous page

FIDELITY OF IMPLEMENTATION

UCSMP Newsletter

Fall 2007–08 Page �

The Secondary Component of UCSMP is interested in issues related to fidelity of implementation, but it approaches the issue somewhat differently than

CEMSE’s FOI Project (see p. 6). Each secondary text is field-tested for a full school year prior to commercial publication. The field test enables UCSMP to obtain information about the feasibility of new approaches, about the effectiveness of new content, and about the quality of the narrative and the problems. The information obtained from the teachers informs the Project of revisions to improve the books prior to commercial publication. In addition, the information gives us insight into the extent to which teachers are able to implement the text as intended. This information is collected from five sources, described here.

Chapter Evaluation Forms. For every chapter, teachers complete an evaluation form. Teachers rate the lesson text and the problems on a five point scale: 5 = Excellent, leave as is; 4 = Good, minor changes needed; 3 = OK, some big changes needed; 2 = Poor, needs major rewrite; and 1 = Disastrous, scrap entirely. In addition, teachers indicate which lessons they covered (including the Self-Test and the SPUR Chapter Review), the number of days spent on a lesson, and the problems assigned. Other questions ask teachers to indicate any supplementary material used, together with a rationale for its use (e.g., more time needed on topic, content required as part of state standards), whether the Chapter Test provided in the Teachers’ Notes was used, whether and how technology was used with the chapter, and whether particular activities or approaches were used. Then, as indicated in the article in this Newsletter on opportunity to learn (see p. 3), Project staff are able to determine the percent of lessons taught and the percent of problems assigned; together these measures give one perspective on fidelity of implementation, the extent to which teachers taught the content of the text.

Classroom Visits. During the course of the year, classes of each UCSMP field-test teacher are visited for two or three consecutive days. These visits enable observers to see how teachers teach the UCSMP secondary curriculum.

Teacher Interviews. As part of the classroom visits, teachers are interviewed about the class, their instructional practices, and how they use particular features of the course. For instance, teachers are asked about: (a) the extent to which instruction

in the classes observed is typical for the class that year, (b) the structure of the class in terms of how students work, (c) their expectations for students to read their text or to write about mathematics, and (d) their expectations for homework. In addition, teachers are asked how they use particular features in the text, such as the Self-Test, the SPUR Chapter Review, or the activities. Finally, teachers are queried about their use of technology and the ways in which the presence and availability of technology impacts their instruction. These questions are important because calculators are provided on loan to teachers in sufficient quantities to be checked out to students.

Teacher Questionnaires. At the beginning as well as the end of the school year, teachers are asked to indicate how important certain practices are to their instruction, such as helping students learn to read their text, reasoning about mathematics, preparing for further study in mathematics, or learning to solve problems. Teachers also indicate the frequency with which they expect students to pose open-ended questions, work in small groups, or explain concepts to others. At the end of the year, both teachers and students are queried about the frequency of reading strategies (e.g., teacher reads aloud, students read silently, students discuss reading) and the frequency of writing opportunities (e.g., to explain problems, to write in journals, to complete projects). Teachers also estimate the percent of their weekly instruction devoted to warm-up, homework review, or introduction of new content. Finally, teachers’ and students’ frequency and type of use related to technology are ascertained.

Opportunity-to-learn forms for posttests. As indicated on page 3, for each item on the posttest, teachers indicate whether they taught or reviewed the content needed for their students to answer the item. Responses to this form provide an indication of the extent to which the content UCSMP chose to assess on the posttest, which was based on the expected content coverage, was taught.

Information from all five of these sources has been extremely beneficial to the Project in making final revisions to the materials and in understanding how the materials are used in classes. Taken together, they have provided Project personnel with a view about the extent to which the materials are implemented in classes as intended. If you have questions about these procedures, please contact Denisse R. Thompson (see p. 11 for contact information).

Fidelity of Implementation in the UCSMP Secondary Component

FIDELITY OF IMPLEMENTATION

UCSMP Newsletter

Fall 2007–08 Page 10

Are you receiving more than one copy of this Newsletter?

Let us know so that we can remove any duplicates from our mailing list. Please send the duplicate label to: UCSMP Newsletter

6030 S. Ellis AvenueChicago, IL 60637

You can also call us at 773-702-1130 or e-mail the record number from the duplicate label to: ucsmp@uchicago.edu.

CSMC

The Center for the Study of Mathematics Curriculum (CSMC), a consortium funded by the National Science Foundation, explores mathematics curriculum—as it is

intended to be, as it is, and as it is learned by students—through research, publications, and conferences.

Those looking forward to the CSMC’s Second International Curriculum Conference (see below) may be interested in the papers presented at the first conference, held in November 2005. This conference focused on the design and development of school mathematics curricula in Asian Pacific Rim countries. The soon-to-be-published proceedings reveal the perspectives

Proceedings of the First CSMC International Conference on Mathematics Curriculum

Coming Soon from the Center for the Study of Mathematics Curriculum:

of education ministries and textbook authors from Japan, Singapore, Korea, and China, and include the responses of U.S.-based speakers and doctoral students. It is sure to interest anybody with questions about the experiences of countries that have adopted a single national mathematics curriculum.

Proceedings of the First CSMC International Conference on Mathematics Curriculum (edited by UCSMP director Zal-man Usiskin) is expected to be available from Information Age Publishing (http://www.infoagepub.com) in January, 2008.

Mark Your Calendar:

“Future Curricular Trends in School Algebra and Geometry” Conference Will be Held May 2-4, 2008

The Center for the Study of Mathematics Curriculum (CSMC) will present a second international conference on the subject “Future Curricular Trends in School Algebra

and Geometry” on May 2–4, 2008, at The Field Museum in downtown Chicago and on The University of Chicago campus.

The conference will feature experts from Australia, Europe, and South America, as well as speakers from the United States, sharing their work and views on school algebra and geometry curriculum and instruction.

Thanks to our funders! Since 1983, UCSMP has received funding from: BP Foundation (formerly the Amoco Foundation) • National Science Foundation • Ford Motor Company • Carnegie Corporation of New York • Stuart Foundation • Verizon Foundation (formerly the General Electric Foundation) • GTE Corporation • Illinois Board of Higher Education • Citicorp/Citibank • Exxon Education Foundation

If you are interested in attending “Future Curricular Trends in School Algebra and Geometry,” please check the CSMC web site, http://mathcurriculumcenter.org for further details and a registration form.

UCSMP Newsletter

Fall 2007–08 Page 11

To learn more about UCSMP and the translations, evaluation reports, and conference proceedings available from UCSMP, visit our web site, http://socialsciences.uchicago.edu/ucsmp.

For questions and general inquiries, send e-mail to ucsmp@uchicago.edu.

To communicate with other users of UCSMP materials, subscribe to our online forums.

UCSMP hosts an e-mail discussion forum for the exchange of ideas related to the use of its elementary and secondary materials. To subscribe, follow the directions at https://listhost.uchicago.edu/mailman/listinfo/ucsmp4um.

The Everyday Mathematics Center hosts an e-mail discussion forum for educators using EM. To subscribe, follow the directions at https://listhost.uchicago.edu/mailman/listinfo/ucsmp-el.This Newsletter was printed on recycled paper with 10% post-

consumer waste.

CONTACT INFORMATION

Online Resources

Publishers of UCSMP Materials

K–6 CurriCulum & TeaCher DevelopmenT

Wright Group/McGraw-Hill • P.O. Box 812960 • Chicago, IL 60681 • (800) 523-2371

K–6 TeaCher DevelopmenT

COMAP • 175 Middlesex Turnpike • Suite 3B • Bedford, MA 01730 • (800) 772-6627

6–12 CurriCulum

2nd Editions

Prentice Hall School Division • 501 Boylston St., Suite 900 • Boston, MA 02116 • (800) 848-95003rd Editions

Wright Group/McGraw-Hill • P.O. Box 812960 • Chicago, IL 60681 • (800) 523-2371

9–12 TeaCher eDuCaTion

Prentice Hall Higher Education Division • 1 Lake St. • Upper Saddle River, NJ 07458 • (800) 350-3693

evaluaTions publisheD by oThers Than uCsmpUMI Dissertation Services • 300 N. Zeeb Rd. • Ann Arbor, MI 48106 • (800) 521-0600

TranslaTions publisheD by oThers Than uCsmpAmerican Mathematical Society • P.O. Box 6248 • Providence, RI 02940 • (800) 556-7774

The UCSMP Newsletter is published twice yearly by the University of Chicago School Mathematics Project, 6030 S. Ellis Ave., Chicago, IL 60637.Editor: Kathleen Andersen

At EdElstonE CEntEr • 6030 S. Ellis Ave. • Chicago, IL 60637Zalman Usiskin, UCSMP Director/Secondary Component Director, (773) 702-1560; z-usiskin@uchicago.eduCarol Siegel, Assistant to the UCSMP Director, (773) 702-9770; cssiegel@uchicago.edu Natalie Jakucyn, Director of Writing, (773) 702-3357; njakucyn@uchicago.eduDenisse R. Thompson, Director of Evaluation, (773) 702-8775; denisse@uchicago.edu; At thE UnivErsity of soUth floridA • College of Education • Tampa FL 33620 • thompson@tempest.coedu.usf.edu

At thE CEntEr for ElEmEntAry mAthEmAtiCs And sCiEnCE EdUCAtion (CEMSE) • 5640 S. Ellis Ave. • Chicago, IL 60637Max Bell, UCSMP Elementary Materials Director, (773) 702-1563; m-bell@uchicago.eduAndy Isaacs, CEMSE Co-Director, (773) 702-9639; aisaacs@uchicago.eduJames McBride, CEMSE Co-Director, (773) 702-2987; jmcbride@uchicago.edu

At ECkhArt hAll • 5734 S. University Ave. • Chicago, IL 60637Izaak Wirszup, Resource Development Director, (773) 667-1967

At ryErson hAll • 1100 E. 58th St. • Chicago, IL 60637Paul Sally, UCSMP Director 1983–87, (773) 702-7388; sally@math.uchicago.edu

At ComAP inC. • 175 Middlesex Turnpike • Suite 3B • Bedford, MA 01730Sheila Sconiers, Elementary Teacher Development Director, (800) 772-6627, ext. 135

At WElls hAll • Michigan State University • East Lansing, MI 48824Sharon Senk, Secondary Component Evaluation Consultant, (517) 353-4691; senk@math.msu.edu

At AnnEnbErg hAll • Northwestern University • Evanston, IL 60208Larry Hedges, Evaluation Consultant, (847) 491-8899; l-hedges@northwestern.edu

UCSMP Directory

The label above is incorrect. Please make the indicated changes.

Please add the name below to your mailing list to receive future mailings.

Please add the name below to your mailing list and send a packet containing a project brochure, recent newsletters, and ordering information.

Name (please print)

Position

Institution

Indicate preferred address below (Home or Work ):

Street

City State Zip

Phone E-mail

Yes, I am using UCSMP materials or ideas, including:

Everyday Mathematics Pre-K K 1 2 3 4 5 6 Everyday TeachingTMStaff Development Kit

Transition Mathematics UCSMP Algebra UCSMP Geometry UCSMP Advanced Algebra

Functions, Statistics, and Trigonometry Precalculus and Discrete Mathematics

Nonprofit OrganizationU.S. Postage

PAID

Chicago, IllinoisPermit No. 9244

UCSMP NewsletterThe University of Chicago School Mathematics Project6030 South Ellis AvenueChicago, IL 60637

For further information . . .