Examining Perceptions of

Success During the Implementation of

an Instructional Policy

Dr. Kathryn OhleUniversity of Alaska Anchorage

Literacy Research AssociationWednesday, December 4th, 2013

Today I will: Describe my approach Explain my methods Share my findings Elaborate on points of interest Discuss potential limitations Outline where I would like to go next Answer any questions

The Context – The Targeted Reading Intervention

(TRI) First grant received in 2005 - both a goal 2 (development) & goal 3

(efficacy) via IES Rural School and Community Trust Grant #R305A10000654-11; latest grant (2010) focuses on goal 3 via IES Grant#R305A040056.

A randomized control trial that uses a dual-level professional development intervention for at-risk K-1 students and classroom teachers.

Teachers learn skills that increase knowledge of early reading skills, improve instructional strategies, and match instruction to informal assessment. They also receive PD in early reading practices during periodic workshops and weekly web-coaching sessions with a coach

Children receive individualized daily instruction in 15-minute lessons that address fluency building, decoding skills, and guided oral reading. Child measures include DIBELS, Woodcock Johnson (5 subtests), &

CTOPP (5 subtests).

Under the direction of Dr. Lynne Vernon Feagans (PI) and Dr. Mary Bratsch-Hines (Program Director)

TRI – 3 Studies’ RCT Outcomes

Vernon-Feagans, L., Gallagher, K., Ginsberg, M., Amendum, S., Kainz, K., Rose, J., and Burchinal, M. (2010).

Learning Disabilities Research to Practice

• Face-to-Face TRI Coaching• N=182; 20 K-1 classroom teachers• K students

Woodcock Word ID effect size= 1.03

• 1st grade no effect & low fidelity

Vernon-Feagans, L., Kainz, K., Amendum, S., Ginsberg, M., Wood, T., & Bock, A. (2012).

Learning Disability Quarterly

• Face-to-Face TRI Coaching• N=276; 34 K-1 classroom teachers• K-1 students

Woodcock Word ID effect size=0.48

Amendum, S.J., Vernon-Feagans, L., & Ginsberg, M.C. (2011)

Elementary School Journal

• Webcam-based TRI Coaching• N=364; 43 K-1 Classroom teachers• K-1 Students

Woodcock Word Attack, Word ID, Passage Comprehension & Spelling of Sounds effect sizes from .4 to .72

The Research Questions for this

Study:• How do stakeholders measure the

“success” of the TRI? • Do stakeholders’ perceptions of a

policy’s success differ within or between groups? How so?

• How do these perceptions of success impact the implementation of the policy?

My Theoretical Approach

Examine TRI as a policy“An unofficial, nongovernmental, or informal

practice that shapes behaviors and outcomes” (Marshall & Gerstl-Pepin, 2005. p.37)

My Theoretical Approach

Pay attention to the perceptions of key

implementers of the policy, within context “Implementers at all levels of the system

effectively negotiate their response, fitting their action to the multiple demands, priorities, and values operating in their environment and the

effective authority of the policy itself” (McLaughlin, 1987b, p.175).

Methods Qualitative methods – selective and open-coding,

constant comparison method Primary source of data – semi-structured interviews

10 teachers, 2 principals, 5 researchers, 1 literacy coach

Secondary source of data – focus groups, field notes, e-mail correspondences

Approach to AnalysisStep 1 – Apply intentional codes to data gathered through interviews

Step 2 – Organize answers into data displays

Step 3 - Look for evidence of codes identified in steps 1 and 2 in secondary data sources; add to data displays

Step 4 – Allow for open-coding to allow for additional patterns or themes to emerge

Step 5 – Identify commonalities and differences within and between groups of stakeholders and explore reasons for variations (informed by literature & critical theory)

Step 6 – Summarize findings

Step 7 – Apply both traditional and critical policy analysis

How do stakeholders measure the “success” of

the TRI?Teachers said, “Student Growth” Standardized test scores

“I just wanted to share some great news - I tested JA today on running records and she passed a level 14!!! She was an 8 in January before we started. I was so excited - along with her mother!!” (Nellie, e-mail communication, March 16, 2012).

“I am sooooooo excited to write and tell you that B (student) went from an “E” to an “I” this nine weeks!!!!!!!!!” (Tonya, e-mail communication, May 21, 2012).

How do stakeholders measure the “success” of

the TRI?Teachers said, “Student Growth” Anecdotal notes related to students’ academic

growth

“He really is moving right along and I've been impressed with his word skills. I still see some weaknesses in fluency and sight words, but I see lots of improvements” (Lauren, e-mail communication, December 21, 2011).

“I thought Josie Anne was going to be retained and never could have gotten her to grade level without TRI!” (Nellie, interview response, August 17, 2012)

How do stakeholders measure the “success” of

the TRI?Teachers said, “Student Growth” Student growth in confidence and motivation

“My kids went from rolling their eyes and saying ‘I can’t read’ to actually reading and beating me to the reading table to do it” (Lauren, interview response, August 15, 2012).

“When the child who hated reading was now asking if he could read to others, I knew it [the TRI] was a success” (Maddie, interview response, August 9, 2012).

How do stakeholders measure the “success” of the TRI?

Teachers said, “Teacher growth” Content knowledge

“On a personal level, it was a wonderful resource for me, to guide me through my first year of first grade. Helped me know how to help my children” (Gloria, interview response, August 13, 2012). Confidence

“Hey, I can really do this” (Pam, interview response, August 13, 2012).

“TRI has changed my perceptions on how long it takes a struggling child to catch up, as I’d always heard it took two years but now I see that it takes shorter” (Maddie, focus group response, August 9, 2012)

How do stakeholders measure the “success” of

the TRI?Researchers said, “Student Growth” Standardized test scores

“if they had made big gains on standardized reading tests, they had hit the gold standard” (Lila, interview response, August 15, 2012)

“I have heard anecdotally from the coaches in team meetings the teachers were successful but some anecdotal evidence shared by the assessors [who administered the battery of tests like the Woodcock-Johnson] indicated some teachers were disheartened. We will just need to see data, mainly the pre and post test scores of the kids as represented through statistics” (Alicia, interview response, August 13, 2012).

How do stakeholders measure the “success” of

the TRI?Principals said, “Student Growth” Standardized test

scores (Dibels & running records)

Literacy coach said, “Teacher efficacy” & “Student Growth” Confidence levels,

often they did it, instructional match

Running record levels, end-of-the-year outcomes

Do stakeholders’ perceptions of a policy’s success differ within or

between groups? How so? Teachers – focus on student achievement scores

(measured through running records), increased levels of student motivation and confidence, and teacher growth and self awareness

Researchers – focus on student achievement scores (DIBELS & WJ scores), some focus on fidelity

Principals – focus on student achievement scores (reading levels and DIBELS)

Coach – focus on teacher capacity and long-term outcomes

How do these perceptions of success impact the

implementation of the policy? Affects motivation & buy-in

“If students, teachers, or administrators believe that the results of an examination are important, the effect is produced by what individuals perceive to be the case” (Madaus, 1999, p.78 as cited in McGee, 2006).

The expectation is that once the teachers see change, they will feel successful and motivated to continue implementing the TRI on a regular basis, as activities that are successful are more likely to be repeated than those that are not (1986). If the teachers do not see a change, especially in student outcomes (i.e. don’t feel successful), there is less inducement for them to change their beliefs, attitudes, or teaching practices (Lamb, Cooper, & Warren, 2007).

Points of Interest Factors that may have impacted

the results: Perceptions of roles Perceptions of real and perceived

lack of control Perceptions of influence Perceptions of responsibility Power dynamics amongst

stakeholders

Limitations Interviewing my own teachers Focus on perceptions means subjective

answers Small sample size limits any type of

generalizability, especially with principals Influences on continued implementation and

teacher buy-in (interviews only after year 1)

Future Directions Add in a Mixed Methods design and compare

standardized assessment results with teacher perceptions

Broaden the group of stakeholders to include parents and students

Revisit after Year Two of implementation Add in coach perceptions in a more

deliberate manner – my influence was obvious!

Questions?Please feel free to contact me at:

kohle@uaa.alaska.edu