implementation of saxon phonics and spelling k web viewimplementation of saxon phonics and spelling...

Running head: IMPLEMENTATION OF SAXON PHONICS

Implementation of Saxon Phonics and Spelling K Evaluation Project Design

Michelle C. Wallace

Virginia Commonwealth University

Author Note

Michelle C. Wallace, Graduate School, School of Education, Virginia Commonwealth

University.

This is an assignment for EDLP 711. The author collaborated with a peer to complete

Field Experience 3: Self-selected Evaluation Project.

IMPLEMENTATION OF SAXON PHONICS 2

Implementation of Saxon Phonics and Spelling K Evaluation Project Design

Elementary schools have the important role of laying the groundwork for creating

successful students. They must teach students many competencies to include social, behavioral,

and instructional skills. One of the most important of these skills is teaching children how to

read. This skill is necessary to promote further success in school as well as in life. Schools are

responsible for teaching students fundamental skills in Pre-Kindergarten and Kindergarten which

will carry them throughout life. One such skill is reading. If students do not learn to read, then

they will never be successful in other subjects which will ultimately impact their success in life.

Saxon Phonics and Spelling – K (2005) has been implemented at ABC Elementary

School since 2009-2010. ABC Elementary School is located in a rural area in Virginia and is

one of three elementary schools within the school division. ABC Elementary School was subject

to a grade reconfiguration for the 2010-2011 school year. ABC Elementary School serves

students in Pre-Kindergarten, Kindergarten, Early Childhood Special Education, and Head Start.

The student population is 49% White, 48% Black, and 3% Hispanic/other, with a Free and

Reduced Lunch rate of 69% (C. Gravely, personal communication, October 5, 2012). An

evaluation of Saxon Phonics and Spelling – K (Saxon Phonics) is needed to evaluate the

implementation of the program at ABC Elementary School. Focus group meetings were held to

gather information regarding the implementation of Saxon Phonics. A responsive evaluation

will be conducted to collect data and prepare a report to the client, ABC Elementary School.

This paper will examine the purpose for the responsive evaluation using a theoretical

lens; reveal the focus for the evaluation questions; and review the evaluation design selection,

evaluation components, and rationale behind the selection. Finally, a logic model will illustrate

the program actions, what the program does, and how the investments will be linked to results.


Purpose

The purpose of the evaluation is to collect information regarding the implementation of

Saxon Phonics at ABC Elementary School. After meeting with staff for focus group meetings at

ABC Elementary School, it became evident that parts of the Saxon Phonics program are being

implemented with fidelity, where other areas still need to be developed. Concerns about the

implementation of the Saxon Phonics program are partly due to emerging technologies, a lag

time in initial training and current needs, and a lack of supplemental fluency materials (Focus

Groups, personal communication, October 5, 2012). Within the values branch, the evaluator’s

role is to be the communicator that promotes dialogue that is meaningful (Mertens & Wilson,

2012). During the evaluation, the evaluators will engage in meaningful conversations with the

stakeholders for effective communication. The evaluators will remain aware of power

differences that exist within the organization and how power may influence the evaluation.

Using leadership roles through the leadership process, the evaluators will bring to light possible

conflicts and provide space for possible discussions about the implementation of Saxon Phonics

(Mertens & Wilson, 2012).

Evaluation Questions

The focus for the evaluation of Saxon Phonics will be the actual implementation versus

the perceived implementation of the Saxon Phonics program in Kindergarten classes at ABC

Elementary School. The evaluand, Saxon Phonics, is a classroom based program; however,

instructional support personnel provide supplemental instruction to reinforce phonemic

awareness and other aspects of the program. The Saxon Phonics program includes (a) the

controlled vocabulary and practice, (b) the annotated bibliography, (c) coding, (d) spelling, (e)

assessments, (f) pacing, (g) handwriting, and (h) reading fluency (Simmons, 2006). The


principal at ABC Elementary is interested in determining how effective the implementation of

the Saxon Phonics program is based on the fact that some components of the program are not

being used. The following questions will be answered as a result of the responsive evaluation.

1. Does the lack of the classroom implementation of all the components of Saxon

Phonics impact the effectiveness of the program?

2. How has the lack of on-going professional development for the Saxon Phonics

program affected the fidelity of the program?

3. How can specific technology enhance the effectiveness of the Saxon Phonics

program?

Design Selection, Components, and Rationale

Evaluation Design Selection

Dividing the world of evaluation into four separate paradigms is one way of organizing

the major influences that have affected the evolution of this transdiscipline (Mertens & Wilson,

2012, p. 133). Fundamentals of the four paradigms will lead the evaluator to different evaluation

approaches. The four paradigms for evaluations are postpositivist, pragmatic, constructivist, and

transformative. The postpositivist paradigm is based on quantitative data. This paradigm was

not chosen by the evaluator because the quantitative data on the evaluand is limited. Even

though the evaluation will use data resources, the purpose of the evaluation is not based on

statistical outcomes. The pragmatic paradigm is based on data deemed important by

stakeholders. This paradigm was not selected by the evaluators because the data used by the

stakeholders will not affect the outcome of the evaluation. The constructivist paradigm looks at

the values and perspectives through different methods. The evaluator chose the constructivist

paradigm and values branch to evaluate the Saxon Phonics program because the evaluators will


use qualitative data to look at varying perceptions and values of the different stakeholders.

Finally, the transformative paradigm focuses on viewpoints of marginalized groups. This

paradigm will not be used by the evaluator since the evaluation does not have a need to look

specifically at the viewpoints of marginalized groups to evaluate the implementation of the

Saxon Phonics program at ABC Elementary.

The constructivist paradigm promotes the responsive evaluation. A responsive

evaluation includes the values which stakeholders find important to the evaluation. These values

drive the development of investigative questions to be answered by the evaluation. A responsive

evaluation allows the evaluators to modify the evaluation based on the needs of the stakeholders.

This allows the evaluators to be responsive throughout the evaluation. The responsive evaluation

will allow stakeholders to explore their perceptions and beliefs about the implementation of the

Saxon Phonics program as they will be active participants in the evaluation process.

Background information gathered through focus group discussions regarding the

implementation of Saxon Phonics led the evaluators to pose evaluation questions using the

responsive evaluation design. The evaluators will utilize qualitative data from observations,

interviews, and documents. Quantitative data will be limited to student data demonstrating

progress of phonological attainment. The responsive evaluation was selected because it utilizes

observations, narrative data, product displays (e.g. student work), and input provided by

stakeholders (Mertens & Wilson, 2012, p. 251). According to Stake (1991), evaluation questions

should reflect the values of what is considered important based on what has been observed by the

evaluator prior to the evaluation of the program (Mertens & Wilson, 2012, p. 151).

The evaluators of this program are internal evaluators. The evaluators are employed by

ABC Elementary School Division and therefore meet the criteria of an internal evaluator. An


external evaluator is not needed at this point in the evaluation process. This evaluation is

formative in nature because the evaluation is based on the current implementation of the Saxon

Phonics program. It is not a summative evaluation because the evaluation will end prior to the

completion of an entire implementation phase.

Components of the Evaluation

The responsive evaluation will provide primarily qualitative data which will help the

evaluators find value in the Saxon Phonics program implementation. Quantitative data will be

limited to student achievement data from the beginning of the year to mid-year. Qualitative data

analysis helps to identify themes in the data (Mertens & Wilson, 2012). Quantitative data

analysis uses statistics to reduce a large data set into more meaningful terms (Mertens & Wilson,

2012). The proposed evaluation will utilize a limited mixed methods approach incorporating

both quantitative and qualitative data, with more reliance on qualitative data. For the purpose of

this evaluation, a mixed methods approach will allow the evaluators to consider quantitative

results during the program implementation and determine through qualitative measures how

much the Saxon Phonics program impacts student achievement.

The evaluation will employ different methods to collect data including observations,

interviews, documents, and student data. Each of the questions posed for the evaluation will be

answered through the data collection process. Question one refers to the classroom

implementation of the Saxon Phonics program, or the lack thereof. The evaluators will use

observations, interviews, and student data to determine how the varying implementation of the

program between classes impacts the effectiveness of the program, if at all. Question two refers

to the professional development for the Saxon Phonics program. The evaluators will use

documents, observations, and interviews to determine how the lack of professional development


since the beginning of the program implementation has had an effect on the fidelity of the

program. Question three refers to how technology could enhance the Saxon Phonics program.

The evaluators will use interviews, observations, and student data to determine the influence that

technology has on the effectiveness of the program. Student data will be limited to measuring

achievement of phonic and literacy attainment. Once the data has been collected, the evaluators

will triangulate the data to determine the degree of implementation of the Saxon Phonics

program. By crossing the data sources, both qualitative and quantitative, the evaluators will get a

clearer image of the impact of the program being evaluated. This process will validate the data

and make it more meaningful to the investigation. The methods for collecting data through

observations, interviews, documents, and student data will be explained.

Observations.

Observations will be conducted in Kindergarten classrooms, Title I Reading classes, and

English Language Learner pull-out sessions. Additional instructional support personnel will be

included in the observations during sessions where students with disabilities are mainstreamed

into the classroom, as well as during sessions when the Speech/Language Pathologist is

integrating the Sounds in Motion program. Sounds in Motion is a program specifically designed

as a kinesthetic learning model to teach the sounds of the alphabet (M. Wallace, personal

communication, November 6, 2012). It is not a component of the Saxon Phonics program but

may have an impact on its implementation. During the observations, a checklist rubric will be

used that includes a description of each of the components of the Saxon Phonics program. The

rubric will help the evaluators determine the level of implementation of each component.

According to Stake (1991), observations are a key factor in responsive evaluations and can


contribute to the overall understanding of the program and can also assist in resolving any issues

identified (Mertens & Wilson, 2012, p. 151).

Interviews.

Interviews will be conducted with stakeholders. The stakeholder groups will represent

(a) teachers, (b) division instructional staff, (c) the building administration, and (d) instructional

support personnel. Interviews will allow for stakeholders to speak freely in regard to their

implementation of the Saxon Phonics program in their classrooms. Interviews will also provide

an opportunity for division and building administration to express their views of the program, as

well as their expectations for program outcomes. The interviews may also lead to suggestions

for strengthening the program or changing the way it is being implemented in classrooms.

Specific questions will be developed by the evaluators to coincide with the components of the

Saxon Phonics program.

Documents.

Program documents, as well as documents kept by the teachers, will be reviewed. The

evaluators will look at teachers’ lesson plan books for documentation of the components used

from the Saxon Phonics program. Classroom materials and student work samples will be

reviewed to determine the level of implementation of the program. A document checklist will be

developed and utilized to collect data on which program components are being implemented

across settings. Saxon Phonics has some optional components. These optional components will

be included in the document review to determine if any teachers are using those components to

enhance their classroom implementation.

Student Data.


Student data will be collected as a part of the program evaluation and will serve as

quantitative data to track the student progress. Six-week assessments on upper and lower case

alphabet recognition and letter sounds will be reviewed. Student achievement will be compared

using baseline data from the beginning of the school year and on mid-year assessments. The

principal informed the evaluators that the baseline data on student knowledge of the upper and

lower case alphabet knowledge was collected (C. Gravely, personal communication, November

6, 2012). Data from the Phonological Awareness and Literacy Screening (PALS) assessment

will be collected from the beginning of the school year and mid-year for students identified as

risk for learning phonological concepts. All Kindergarten students took the PALS assessment at

the beginning of the year, but only students who were identified as at-risk by the pre-assessment

will be tested using the PALS mid-year assessment (C. Gravely, personal communication,

November 6, 2012). The alphabet knowledge and PALS data will be reviewed by classroom to

determine the impact of the implementation of Saxon Phonics by classroom and to identify

whether or not there is an effect on student achievement. Due to the time constraints of this

evaluation, mid-year data will be used to measure student achievement. If the evaluation could

extend to the end of the school year, then end of year data would also be included.

Rationale

Various other evaluation methods could have been selected to use to determine the

effectiveness of the implementation of Saxon Phonics at ABC Elementary School. Using the

responsive model for the evaluation will fuel conversations about the actual implementation of

the Saxon Phonics program in the Kindergarten classrooms and supplemented by the

instructional support personnel. As issues arise during the evaluation, the evaluators can be

responsive to those issues at that time. “Conducting an evaluation that is responsive to the needs


of the program stakeholders. . .will ensure that what stakeholders have learned through their

active participation is meaningful and is more likely to be used for improving and enhancing

programs” (Mertens & Wilson, 2012, p. 151).

Logic Model

The evaluation of Saxon Phonics will rely on certain inputs, outputs, and resources which

will provide anticipated outcomes based on a regimented timeline for completion of the

evaluation and reporting. The inputs will include proposed data sources from the Saxon

curriculum, observations, interviews, program documents, and student data. For the data

analysis, observations and interviews will provide a wealth of information to help determine the

level of fidelity in the program implementation. Student data will be instrumental in helping the

evaluators determine the effectiveness of the Saxon Phonics program when compared to the

actual implementation of the components. Although it is not the intent of the responsive

evaluation format, data may allow the evaluation to slide into the transformative paradigm.

Depending on the student demographic data provided by the building administrator, the

evaluators may also be able to determine if specific Gap groups as identified by the federal

Annual Measureable Objectives (AMOs) are determined to be a greater need for phonics

interventions. Gap groups are described by three categories: Gap group 1 represents students

with disabilities, students who are English language learners, students who are economically

disadvantaged; Gap group 2 represents African-American students, not from Hispanic origin;

and Gap group 3 represents Hispanic students of one or more races (Virginia Department of

Education, 2012). The activities to conduct the evaluation and collect data will follow a specific

timeline and will require specific resources.


The timeline for the evaluation will fall between the months of December 2012 and

March 2013. During December, the evaluators will present the evaluation proposal to the client.

Data collection tools will be developed during this time and approved by the client. Beginning

in January, evaluators will begin the review of student data (baseline and mid-year). Also in

January, interviews and classroom observations will be conducted of the Kindergarten

classrooms. Evaluators will also interview division and building administration staff. During

February, a mid-term formative report will be provided by the evaluators to the client.

Classroom observations and interviews will be conducted with the instructional support

personnel. In March, the evaluators will present a summative report of findings to the client.

Specific dates will be determined upon the client accepting the evaluation proposal. The client

will need to agree to provide required resources for the evaluation. Required resources include:

a room for evaluators to work and conduct interviews; classroom coverage for teachers to

participate in interviews; access to student data agreed upon by the client and the evaluators; and

time for classroom observations to occur. It is suggested that $250 be budgeted by the client for

incidentals such as snacks for the interviewees and ink for printing purposes. Since the

evaluators are internal evaluators, there will be no cost for their time. The evaluators will

provide tangible resources such as a recorder for interviews, checklists to be used during

observations and document reviews, and other data collection tools.

It is anticipated that the evaluation will provide information based on data triangulation

which will help ABC Elementary School determine short and long term implications. In the

short term, the evaluation will determine the implementation of the Saxon Phonics program, the

level of support offered to teachers, and the effectiveness of the program. Long term outcomes

may implicate future needs for a more successful implementation of the program, determine the


most effective components of the program, and possibly indicate which Gap groups benefit or

suffer the most from the implementation of the program. A logic model is used to illustrate the

sequence of the Saxon Phonics program actions, what the program does, and how the

investments in it will be linked to its results. As demonstrated in Figure 1, the evaluation process

is impacted by certain assumptions and external factors which may influence the data collection

process for the evaluation of the implementation of Saxon Phonics.

Logic Model

Inputs Outputs Outcomes -- ImpactActivities Participation Short Long

-Saxon Curriculum

-Observations

-Interviews

-Program Documents

-Student Data *PALS *6 Week assessments *Pre and post assessments

-Observation of classrooms

-Individual interviews

-Pre- and Post-assessments for students

-Review of program documents

-Presentation of mid-term and final reports

-Students

-Teachers

-Building and district administration

-Two internal evaluators

-Determining the implementation of Saxon Phonics in classrooms

-Determine level of support for Saxon Phonics offered by instructional support personnel

-Determine the effectiveness of the implementation

-Determine needs for future implementation

-Determine which components are most effective for student achievement

-Determining the needs of Gap groups as defined by AMOs

Assumptions-The client will provide time for the teachers to participate in the interviews.-The client will provide access to student data needed for the evaluation.-The responsive evaluation will be sufficient to answer the questions regarding the implementation of the program.

External Factors-Time will be given for the interviews to take place.-Resources will be available for classroom coverage during interviews.-A schedule will allow for classroom observations of all Kindergarten classrooms and interactions with the instructional support personnel.

Figure 1. The logic model for the evaluation of Saxon Phonics demonstrates the input and output components of the evaluation of the program. Also considered are the assumptions and external factors which influence the outcomes of the evaluation.

A meta evaluation will be conducted throughout the entire process of the evaluation to

determine the quality of the evaluation process. Mertens and Wilson (2012) suggest that the


meta evaluation involve the stakeholders to discuss the evaluation design and process. Also,

methodologies should be discussed about the data collection strategies and how information will

be disseminated to the client and other stakeholders (Mertens & Wilson, 2012). During the

evaluation planning process, the evaluators considered program evaluation standards and have

determined that the evaluation meets criteria for utility, feasibility, propriety, and accuracy.

Table 1 depicts the evaluation standards which have been met thus far in the evaluation planning

phase or are planned to be met during the evaluation action phase.

Table 1

Summary of The Program Evaluation Standards Met for the Evaluation Proposal for the Implementation of Saxon Phonics at ABC Elementary School

Standard Descriptor

Utility 1 Evaluator Credibility

Utility 2 Attention to Stakeholders

Utility 3 Negotiated Purposes

Utility 7 Timely and Appropriate Communication and Reporting

Feasibility 1 Practical Procedures

Feasibility 3 Resource Use

Feasibility 4 Project Management

Propriety 1 Responsive and Inclusive Orientation

Propriety 2 Formal Agreements

Propriety 3 Human Rights and Respect

Propriety 4 Clarity and Balance

Accuracy 1 Trustworthy Conclusions and Decisions

Accuracy 2 Valid Information

Accuracy 3 Reliable Information

Accuracy 4 Explicit Evaluand and Context Descriptions

Accuracy 5 Sound Qualitative and Quantitative Methods

Accuracy 8 Valid Communication and Reporting


Resources

Mertens, D. M., & Wilson, A. T. (2012). Program evaluation theory and practice: A

comprehensive guide. New York, NY: The Guilford Press.

Saxon Publishers. (2005). Scientific research base for Saxon phonics and spelling K-3:

Foundational research and program efficacy studies. Retrieved from

http://saxonpublishers.hmhco.com/HA/correlations/pdf/s/saxon_phonics_research.pdf

Simmons, L. (2006). Saxon phonics and spelling K: Instructional overview. Austin: Saxon

Publishers, Inc.

Virginia Department of Education (2012). ESEA flexibility. Retrieved from

http://www.doe.virginia.gov/federal_programs/esea/flexibility/index.shtml

http://www.doe.virginia.gov/federal_programs/esea/flexibility/index.shtml