Annotated Bibliography 1

Running head: ANNOTATED BIBLIOGRAPHY

Annotated Bibliography

Glenn E. Malone

Washington State University

Annotated Bibliography 2

Introduction

The purpose of this review is to begin my exploration of a possible topic for my

dissertation. I looked at the variety of research on the issues, successes and solutions that

technology integration and specifically online learning has faced and the impact on K-12 systems

in the United States. I am particularly interested in the role of the superintendent in leading a

district through this fast-paced revolution.

My search included the terms Leadership, Online Learning, and Superintendent. I was

unsuccessful in finding many articles specifically on a Superintendent’s role but I did find a

promising and growing body of research on online learning effectiveness.

Patterns emerged in the literature and for this project I have grouped the articles into

three categories: Technology Integration Effectiveness, Online Learning Effectiveness, and

Leadership Policy Implications.

Annotated Bibliography 3

Technology Integration Effectiveness

Becker, H. J., & Ravitz, J. L. (2001). Computer use by teachers: Are Cuban's predictions

correct? Paper presented at the 2001 Annual Meeting of the American Educational

Research Association, Seattle, WA.

In his book Oversold & Underused: Computers in the Classroom, Larry Cuban asserts

that computers have had a minimal impact in classrooms and are minimally used by teachers. In

this paper Becker and Ravitz examine Cuban’s claims and present data from their national

survey of 4,100 teachers from three sample populations. Statistically, their work supports

Cuban’s claims; however further disaggregation of their data revels that when examining

teachers matching a particular profile, Cuban’s findings do not stand true. It was not surprising

to find that, if the teacher had at least five computers in their classroom, takes a leadership role

with peers, had at least moderate computer expertise and was in the top quartile in terms of

constructivist philosophy, computers played a significant role in their classrooms. With teachers

that fit this profile, 75% had their students using word processing software during class ten plus

times a day, and these teachers were roughly three times as likely as all other teachers to do so.

Conversely, one of their strongest and most prevalent findings show that teachers who

shun computers are also the ones who seem to be the most traditional in their philosophy; “that is

those who believe their role is to transmit to students an externally mandated curriculum by

means of a highly controlled pedagogy” (Becker & Ravitz, 2002, p. 9). Unfortunately, only 10%

of the teachers surveyed met their criteria of exemplary computer users. This seems to point to a

need for more professional development in this area.

Annotated Bibliography 4

The purpose of their study focused on the use of computers in the classroom and the

reasons that prohibited the use of computers. Although Becker and Ravitz did not mention

computer labs specifically, their study lends credibility to my long held belief that computers

should be located in classrooms before labs. Their work confirms the need for leaders to

distribute computers to classrooms in clusters of five to eight computers if they want them to be

used. The missing piece for me was while only addressing how often computers are used, the

paper does not address the merits or educational value of computer use as related to student

achievement.

Earle, R.S. (2002). The integration of instructional technology into public education: Promises

and challenges. Educational Technology, 42(1), 5-13.

Earle declares from the start that there is no guarantee that technology will improve

student achievement. He asserts that research on the effects of technology on student

achievement offer mixed results, with fewer than half of the teachers in a nationwide survey

using computers for instructional purposes. His work points to the differing priorities that reveal

teachers would rather see additional funding used to increase the number of computers in the

classroom while superintendents believe the use of funds on teacher training would best improve

computer effectiveness. Earle cites several educational sources that bring insight into the

challenges researchers face such as claims of computer investments in schools as “glitzy toys”

and “bogus stuff”. Earle (2002) emphasizes the value of the teacher and says, “Technology

generates a glut of information, but has no particular pedagogical wisdom" (p. 9).

Annotated Bibliography 5

Earle ends his article with what he calls first order and second order barriers to successful

technology integration. His first order barriers are extrinsic; such as access, support, resources,

and time. The second order barriers he describes are intrinsic; such as attitudes, beliefs,

practices, and resistance; Earle emphasizes the need to address second order barriers first. He

cautions that technology can and does cause a distraction from what matters most, effective

learning and good teaching. He acknowledges that a well trained and effective teacher is

preferable to the most advanced technology. In his final reflection he lists six factors critical to

supporting measurable gains in educational outcomes: leadership, solid educational objectives,

professional development, adequate technology resources, time, and evaluation.

Earle clearly makes a case that technology holds great promise for use in the classroom

while he cautions that we not stray from solid teaching. His work identifies some of the

challenges that I have encountered in my work as a principal and is congruent with my

experiences as a Director of Technology.

Mulkeen, A. (2006). ICT in schools: What is of educational value? In L. Hin (Ed.), Handbook of

Research on Literacy in Technology at the K-12 Level (pp. 73-93). Hershey, PA:

Information Science Reference.

This is an exceptional article that gives a well founded argument for the use of

Information and Communication Technology (ITC). The term ITC is common in Australia,

England and New Zealand and it refers to what we typically call instructional or educational

technology in the states.

Annotated Bibliography 6

Mulkeen asserts that many times teachers throw out effective teaching activities for lesser

experiences with computers. He further argues that computer use in classrooms is often not very

challenging and may be less valuable than what it replaces.

To measure the effectiveness of ICT Mulkeen presents a simplistic but powerful tool he

calls the value grid. He uses the grid to illustrate the interplay between the complexities of

thinking and curricular relevance all the while integrating ICT.

Mulkeen suggests that instead of gullible acceptance of ICT, we need to be much more

demanding of it. He advocates the need to recognize that some uses of ICT are actually contra-

educational, because they displace more valuable activities. I particularly appreciated his

recognition that getting higher-order benefits is not a question of technology, but one of

pedagogy.

The results of Mulkeen’s research illustrate the need for educators to be more selective in

the strategies they employ. He recognizes the techno-lust factor that sometimes blinds well

intentioned teachers from seeing the error of their lesson plan selection.

Schacter, J. (1999). The impact of educational technology on student achievement: What the

most current research has to say. Retrieved July 10, 2009, from Milken Exchange on

Educational Technology Web site: http://www.mff.org/pubs/ME161.pdf

This article is an analysis of over 700 empirical research studies including a study of the

entire state of West Virginia, and a national sample of fourth and eighth grade students. Schacter

found that students with access to: computer assisted instruction, integrated learning systems

technology, simulations and software that teaches higher order thinking, collaborative networked

Annotated Bibliography 7

technologies or design and programming technologies show positive gains in achievement on

researcher constructed tests, standardized tests and national tests. On the other hand they also

point to evidence in some of these same studies that learning technology is less effective or

ineffective when the learning objectives are unclear and the focus of the technology use is

scattered. The author suggests that one of the continuing difficulties is that many teachers think

about the technology first and the education later.

Schacter’s review shows that in all the studies the positive impact by far prevails over the

negative impact. When there was a negative impact it was either the wrong approach or the

teaching style.  All of the studies show an improvement on students’ attitude towards learning

which helps motivate underachieving students. This work seems to suggest that technology,

when used properly, is a great tool to engage and empower students to heighten their attitude

towards learning and allow them to succeed. 

From my review of the research, most of the big name reports and researchers where

included in this project and Schacter’s findings will be very helpful to leaders trying to make

decisions about educational investments. This is one of my better resources on this topic of

technology integration effectiveness.

Waxman, H., Lin, M., & Michko, G. (2003) A meta-analysis of the effectiveness of teaching and

learning with technology and student outcomes. Retrieved on July 9, 2009, from

Learning Point Associates Web site: http://www.learningpt.org/pdfs/tech/preservice.pdf

The purpose of this study is to provide a quantitative synthesis of recent research on the

effects of teaching and learning on student outcomes. To accomplish this task, Waxman

Annotated Bibliography 8

examined 42 studies that contained a combined sample size of 7,000 students. The statistical

data he examined indicate that teaching and learning with technology has a small, positive and

significant effect on student outcomes when compared to traditional instruction. On the other

hand, the studies that contained behavioral outcomes indicate that technology had a small,

negative effect on students’ behavioral outcomes. In statistics, a meta-analysis combines the

results of several studies that address a set of related research hypotheses; one area in which

there have not been many meta-analyses and systematic reviews of the research is how teaching

and learning with technology impacts student outcomes.

This research is important because some studies have found that technology can change

teachers’ pedagogic practices from a teacher-centered or teacher-directed model to a more

student-centered classroom where students work cooperatively, have opportunities to make

choices, and play an active role in their learning.

It was surprising for me to learn that while the impact of technology integration on

student outcomes was significantly positive some studies showed a negative, although small

effect on behavioral outcomes. This was the opposite of what I expected to find and contrary to

other, more recent research on the same topic.

Annotated Bibliography 9

Online Learning Effectiveness

Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallett, P.A.,

Fiset, M. and Huang, B. (2004). How does distance education compare with classroom

instruction? A meta-analysis of the empirical literature. Review of Educational Research.

74 (3), 379–439.

Bernard et al. conducted a meta-analysis of the comparative distance education literature

between 1985 and 2002. In total, 232 studies containing 688 independent achievement, attitude,

and retention outcomes were analyzed. Overall results indicated effect sizes of essentially zero on

all three measures and wide variability. This suggests that many applications of distance

education outperform their classroom counterparts and that many perform more poorly. Dividing

achievement outcomes into synchronous and asynchronous forms of distance education produced

a somewhat different impression. In general, mean achievement effect sizes for synchronous

applications favored classroom instruction, while effect sizes for asynchronous applications

favored distance education. However, significant differences remained in each subset.

This meta-analysis is outdated in terms of the fast-paced nature of the growth and

development in online learning. This meta-analysis is one of many that show no significant

difference in traditional and online learning.

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-

based practices in online learning: A meta-analysis and review of online learning studies.

Retrieved July 9, 2009, from http://www.ed.gov/rschstat/eval/tech/evidence-based-

practices/finalreport.pdf

Annotated Bibliography 10

This report is the result of a meta-analysis involving research published from 1996 to

2008, in which the authors filtered through more than 1,100 empirical studies on online learning.

The study included only objective measures of learning from studies with controlled designs that

met the researchers' minimum quality standards. It included research on both K-12 and post-

secondary education.

The studies of more recent online instruction included in this meta-analysis found

that, on average, online learning, at the post-secondary level, is not just as good as but more

effective than conventional face-to-face instruction. Hearing this, U.S. Secretary of Education

Arne Duncan said, “This new report reinforces that effective teachers need to incorporate digital

content into everyday classes and consider open-source learning management systems, which

have proven cost effective in school districts and colleges nationwide. We must take advantage

of this historic opportunity to use American Recovery and Reinvestment Act funds to bring

broadband access and online learning to more communities."

The report also mentions the recent explosion of online learners; reporting the number of

K-12 public school students enrolling in a technology-based distance education course grew by

65 percent in the two years from 2002-03 to 2004-05. The report also indicates that more than a

million K-12 students, nation-wide took online courses in school year 2007-08.

This report provides the strongest evidence to date for the consideration of online

learning to be included in public schools.

Annotated Bibliography 11

Leadership Policy Implications

Gonzales, A. & Barbour, M. (2008). U.S. virtual school trial period and course completion

policy study. In G. Richards (Ed.), Proceedings of World Conference on E-Learning in

Corporate, Government, Healthcare, and Higher Education 2008 (pp. 2653-2660).

Chesapeake, VA: AACE.

In virtual school settings, student attrition is believed to be a significant issue. Gonzales

& Barbour found attrition rate estimates range from 12% to 50%. These estimates vary greatly

due to the differences in the way each virtual school calculates its student attrition rate. The

purpose of this study was to explore these policy variations in how United States K-12 virtual

schools calculate completion rates.

Gonzales & Barbour sent online surveys to 159 U.S. virtual schools in various

geographic regions across the nation and they received responses from 86 schools. Of those that

responded, 68.6% had trial periods that varied from one day to 185 days. They found that this

variation dramatically impacts the reported attrition rate. Because of the competition for

students, all schools want to publish a low attrition rate. If they give students a longer trail

period then a student dropping during the trial is less likely to negatively impact the school’s

attrition rate. In some cases this manipulation of the data serves to distort the reality of the

published attrition rates.

Course completion definitions between virtual schools also varied considerably from

remaining in the course irrespective of the final grade to receiving an A-, considered a passing

grade. This too will impact the published attrition rates.

Annotated Bibliography 12

Gonzales & Barbour bring to light the need for virtual schools to adopt multiple measures

for calculating student attrition to allow meaningful comparisons between virtual and also with

brick and mortar schools.

Picciano, A. G. & Seaman, J. (2009). K-12 online learning: A 2008 follow-up of the survey of

U.S. school district administrators. Hunter College and Graduate Center of the City

University of New York, Olin and Babson Colleges. The Sloan Consortium. Retrieved on

July 10, 2009 from http://www.sloan-c.org/publications/survey/K-12_06.asp.

This report builds off an earlier one from two years before where these same authors

chronicled the emerging landscape of online learning in K-12 education through a survey that

reached district administrators in a range of representative districts: rural, town, suburban, and

urban.

In 2007, the Sloan Consortium issued a report on the extent and nature of online learning

in K-12 schools. It was based on a national survey of school district administrators during the

2005-2006 academic year. It was one of the first studies to collect data on and compare fully

online and blended learning (part online and part traditional face-to-face instruction) in K-12

schools. The purpose of this current study is to replicate the original study in order to

substantiate its findings and to examine what if any changes occurred in online learning in K-12

school districts. The current study was conducted two years later and was based on a national

survey of school district administrators during the 2007-2008 academic year.

The survey found that 75% of public school districts offer online or blended courses.

Most of these districts anticipated their online offerings will grow, and the biggest growth

Annotated Bibliography 13

appears to be in blended learning. Considering the findings of Mean (2009) which support the

effectiveness of the blended approach, this makes good sense.

According to Picciano & Seaman’s numbers, enrollments in online courses grew 47% from

2005-06 to 2007-08. The survey also shows that online courses are meeting the needs of a range

of students; from those who need credit recovery to those desiring advanced courses. To meet

this need districts are relying on multiple online learning providers. Specifically online courses

represent an absolute lifeline for small rural school districts. Not only do they help them offer

enrichment courses, they also help them offer basic, core courses, too, that they would otherwise

not be able to offer.

Shuldman, Mitchell. (2004). Superintendent conceptions of institutional conditions that impact

teacher technology integration. Journal of Research on Technology in Education, 36(1),

319-343.

The study examined the conceptions of three New Hampshire superintendents as to the

institutional conditions they believe impact teachers' ability to integrate technology. Common

thinking was found. They agree if integration is to be successful, teachers need a better

understanding of how these technologies can be applied in instruction. They believe that multiple

levels of leadership are essential, and they see lack of time and the public's hesitation to spend

public monies on building teacher capacity as obstacles.

The Superintendents’ gave a similar list of obstacles that impede a teachers' ability to

adopt and integrate technology into their teaching such as lack of time, expertise, access,

resources and support. This supports the notion that technology's greatest impact on student

Annotated Bibliography 14

learning appears only after teachers have sufficient skills coupled with an understanding of how

various technologies can be used as cognitive tools, and are able to weave technology

experiences into their daily practice. This more robust level of understanding comes over time.

Shuldman also found that administrators do not appear prepared for their emerging role

in technology and their lack of understanding and resources sometimes creates barriers to change

and improvement. In many districts, superintendents have remained withdrawn from the

technology discussions, leaving to staff the leadership roles of planning and implementing

technology. However, the people who will ultimately implement an innovation must possess

sufficient knowledge and skills to do the job.

Successful practices implemented by superintendents in their districts include the

following:

1. Active participation on the district's technology committee, as chair person,

engagement in active discussions with tech personnel to smooth out and clarify

lines of responsibility, authority, and overlapping interests.

2. The creation of site-based technology committees to ensure active engagement by

principals.

3. Redefinition of technology teacher positions and the creation of technology

curriculum coordinator positions.

4. Use of the persuasive power of the superintendency to help a tech coordinator

evolve middle school reading specialists into full-time technology integrators and

to infuse the district technology education faculty with technology skills.

5. Assist in establishment of free evening technology classes for adults.

Annotated Bibliography 15

6. Facilitate a project that has middle school students teaching email tech to senior

citizens.

7. Actively engage other locally elected civic boards into the district tech discussion

to build broader community support.

Shuldman found that superintendents needed to have principals take the lead in ensuring

that all teachers in their building work towards adopting technology. A superintendent must get

principals to make sure that teachers included technology goals in their individual professional

development plans.

Annotated Bibliography 16

Conclusion

I see tremendous potential for continued research regarding the dramatic, disruptive

impact of online learning on the K-12 systems. However, it appears that my challenge will be in

winnowing a little further to a more specific topic.

The literature provides valid reasons to pursue online learning for our students. The

effectiveness of technology integration (Mulkeen, 2006) and online learning (Bernard, 2004) has

given credibility to these methods as viable and productive methods for learning. Most

encouraging was the recently released research that found online learning in tandem with

traditional learning was more effective than either alone (Means, 2009).

I was discouraged to find very little research on the Superintendent’s role in leading

district efforts to begin online learning initiatives. Shuldman (2004) gave compelling evidence

as to why this is an important, untapped area; fertile for new research.