document resume ed 425 191 · 2014-05-19 · document resume. tm 029 255. wenglinsky, harold does...

ED 425 191

AUTHORTITLE

INSTITUTION

PUB DATENOTEAVAILABLE FROM

PUB TYPEEDRS PRICEDESCRIPTORS

IDENTIFIERS

ABSTRACT

DOCUMENT RESUME

TM 029 255

Wenglinsky, HaroldDoes It Compute? The Relationship between EducationalTechnology and Student Achievement in Mathematics.Educational Testing Service, Princeton, NJ. PolicyInformation Center.1998-09-0041p.

Policy Information Center, Mail Stop 04-R, EducationalTesting Service, Rosedale Road, Princeton, NJ 08541-0001;Tel: 609-734-5694; e-mail: [email protected]; Website:http://www.ets.org/research/pic ($9.50).Reports Research (143)MF01/PCO2 Plus Postage.Access to Information; *Computer Uses in Education;*Educational Technology; *Elementary School Students; Grade4; Grade 8; Intermediate Grades; Junior High Schools;*Mathematics Achievement; Mathematics Education; MiddleSchools; *Outcomes of Education; Poverty; ProfessionalDevelopment; Rural Schools; Suburban Schools; *TechnologicalAdvancement; Urban SchoolsMiddle School Students; National Assessment of EducationalProgress

This report presents findings from a national study of therelationship between different uses of educational technology and variouseducational outcomes. Data were drawn from the 1996 National Assessment ofEducational Progress (NAEP) in mathematics, consisting of national samples of6,227 fourth graders and 7,146 eighth graders. Data include information onthe frequency of computer use for mathematics in school, access to computersat home and in school, professional development of mathematics teachers incomputer use, and the kinds of instructional uses of computers in theschools. The study finds that the greatest inequities in computer use are notin how often they are used, but in the ways in which they are used. Poor,urban, and rural students are less likely to be exposed to higher order usesof computers than nonpoor and suburban students. For both fourth and eighthgrades, teachers of urban and rural students are less likely to have hadprofessional development in technology than suburban teachers. There were fewdifferences in the frequency of school computer use in either grade, althoughblack fourth graders reported more frequent use than white fourth graders.Yet for both grades, black students were less likely to have a computer atschool. In essence, the study found that technology could matter, but thatthis depended on how it was used. The size of the relationship between thevarious positive uses of technology and academic achievement was negligiblefor fourth graders, but substantial for eighth graders. Taken together,findings indicate that computers are neither a cure-all for problems facingthe schools nor mere fads without impact on student learning. When usedproperly, computers may serve as important tools for improving studentproficiency in mathematics and the overall learning environment of theschool. An appendix discusses how the study was conducted. (Contains 2tables, 14 figures, and 23 references.) (SLD)

CP)

1.r)

0 V111 I

a

U S DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

1.0d'<lis document has been reproduced as

CNI

received from the person or organizationoriginating it

Cif 0 Minor changes have been made to

CNIimprove reproduction quality

Points of view or opinions stated in thisdocument do not necessarily representofficial OERI posalon or policy

a

1

PERMISSION TO REPRODUCE AND

DISSEMINATE THISMATERIAL HAS

BEEN GRANTED BY

e_y

TO THE EDUCATIONALRESOURCES

INFORMATION CENTER(ERIC)

This report was writtenby Harold Wenglinskyof the ETS PolicyInformation Center.

The views expressed in thisreport are those of theauthor and do not necessar-ily reflect the views of theofficers and trustees ofEducational Testing Service.

Additional copies of this reportcan be ordered for $9.50 (prepaid)from:

Policy Information CenterMail Stop 04-REducational Testing ServiceRosedale RoadPrinceton, NJ 08541-0001(609) 734-5694Internet - [email protected] can also be downloadedfrom www.ets.org/research/pic

Copyright © 1998 by EducationalTesting Service. All rights reserved.Educational Testing Service is anAffirmative Action/Equal Opportu-nity Employer. The ETS logo is aregistered trademark of Educa-tional Testing Service. Themodernized ETS logo is atrademark of EducationalTesting Service.

September 1998

CONTENTS

Preface 2

Acknowledgments 2

Summary of Findings 3

Introduction 5

Chapter One: The Debate on Technology's Effectiveness 7

The Case for Technology 7

The Case against Technology 8

The Evidence on Technology's Effectiveness 9

Chapter Two: Patterns of Educational Technology Use 12

Data and Measures 12

Access to School Computers 14

Access to Home Computers 16

Teacher Preparation 19

Types of Computer Use 21

Chapter Three: Modeling Technology's Effectiveness 25

How the Study Was Conducted 25

Model Results: Fourth Grade 26

Model Results: Eighth Grade 28

The Size of the Relationships 29

Conclusion 32

Interpreting the Results 32

Knowing More 32

Implications of Findings 33

References 35

Appendix: How the Study Was Conducted 37

3

PREFACE

In this age of technologi-cal imperative, we do itsimply because it can bedone. Massive efforts areunderway to converttraditional teaching tosomething that can bedelivered via computer.Measuring success hasbeen a simple matter:count the number ofcomputers, divide that bythe number of students,and report how the ratioof computers to studentshas advancedand it isalways advancing. Thenclose the report lament-ing that we don't knowmuch about the softwarebeing used on thesecomputers, we don'tknow how many arebehind locked doors, wedon't know how manyare broken, and we don'tknow how many teach-ers really know how touse them (there are noassessments of teachercapability here).

The Policy Informa-tion Center reported inMay of 1997 what we didknow at that time, inComputers and Class-rooms. We said then thatthe available data, andthe research performedon it, did not tell uswhether computer-delivered instructionactually improved perfor-mance. This report takesat least the first step indetermining whethercomputer use is making

2

a difference in math-ematics, and what kindof computer use haswhat kind of effect,on which groupsof students.

This study uses anational database, the1996 National Assess-ment of EducationalProgress, and advancedanalysis techniques, toisolate the effects of thecomputer from themyriad other factorsinvolved in studentachievement. The studywas suggested to usby Education Week,and its author, HaroldWenglinsky, of the ETSPolicy InformationCenter, has collaboratedwith the staff of Educa-tion Week throughoutthe analysis and writingphases of the report.

In addition to tellingus what he found,Harold Wenglinskyalso tells us whatfurther research mustbe conducted to learnmore, and give thenation specific guidancein its efforts to raiseeducational achieve-ment through greateruse of technology.

Paul E. BartonDirectorPolicy Information Center

ACKNOWLIEDGMIENTS

This project involved aclose collaborationbetween myself and themany people workingon Technology Counts, ayearbook on educationaltechnology funded bythe Milken FamilyFoundation and pub-lished by EducationWeek. Indeed, the studywas inspired by aconcern at TechnologyCounts that there was alack of national informa-tion on the relationshipbetween various aspectsof educational technol-ogy and academicachievement. I amindebted to the Technol-ogy Counts staff for theirongoing and timelyfeedback on the project,particularly Craig Jerald,Virginia Edwards, andJeffrey Archer. Such acollaboration betweenresearchers and journal-ists, sometimes referredto as computer-assistedjournalism, is a criticaltool for making availableto the public the fruits ofstatistical analyses oflarge-scale databases.

Many others alsocontributed to thisproject. Stephen Gorman,John Barone, JohnMazzeo, and AlfredRogers facilitated accessto the data. Henry JayBecker, Ellen Mandinach,and Paul Barton pro-vided thorough andtimely reviews of the

4

draft report. RichardColey assisted in pro-ducing the figures. CarlaCooper provided desk-top publishing services.Kirsty Brown did theediting, Kelly Gibsonwas the cover designer,and Jim Chewningcoordinated production.Any errors of fact orinterpretation includedin this report, however,are my responsibility.

SUMMARY OF FINDINGS

This report presentsfindings from a nationalstudy of the relationshipbetween differentuses of educationaltechnology and variouseducational outcomes.Data were drawn fromthe 1996 NationalAssessment of Educa-tional Progress (NAEP)in mathematics, con-sisting of nationalsamples of 6,227 fourth-graders and 7,146eighth-graders. NAEPincludes informationon educational tech-nology, including:

frequency of com-puter use for math-ematics in the school;access to computersand frequency ofcomputer use inthe home;professional develop-ment of mathematicsteachers in computeruse;kinds of instructionaluses of computers bymathematics teachersand their students

NAEP also includesvarious outcomes ofeducation that are highlyvalued by policymakersand the public at large,including: academicachievement in math-ematics as measuredby a standardizedtest; and the socialenvironment of the

school, including studenttardiness and absentee-ism, teacher absentee-ism, teacher morale,and student regard forschool property.

The study firstcompared the informa-tion about educationaltechnology amongdifferent groups ofstudents to discover anypossible inequities intechnology use. It foundthat the greatest inequi-ties did not lie in howoften computers wereused, but in how theywere used.

For eighth-gradersonly, Black studentswere less likely to beexposed to higher-order uses of comput-ers and more likely tobe exposed to lower-order uses than Whitestudents. Similarly,poor, urban, and ruralstudents were lesslikely to be exposedto higher-order usesthan non-poor andsuburban students.

For both eighth- andfourth-graders, urbanand rural studentswere less likely tohave mathematicsteachers who hadreceived professionaldevelopment intechnology over thelast five years thansuburban students.

5

For eighth-graders,poor students wereless likely to havemathematics teacherswho had receivedsuch professionaldevelopment thannon-poor students.

There was littledifference in thefrequency of schoolcomputer use in eithergrade, except thatBlack fourth-gradersreported more fre-quent school com-puter use than Whitefourth-graders.

For both eighth- andfourth-graders, Blackstudents were lesslikely to have accessto a home computerthan White students.Similarly, poor, urban,and rural studentswere less likely tohave access to ahome computerthan non-poor andsuburban students.

The study thenrelated these differentaspects of technologyto the educationaloutcomes, taking intoaccount differences instudent bodies and theother characteristics ofthe schools being ana-lyzed. In essence, thestudy found that technol-ogy could matter, butthat this depended upon

how it was used. Foreighth-graders, the studyfound that:

Teacher's professionaldevelopment intechnology and theuse of computers toteach higher-orderthinking skills wereboth positively relatedto academic achieve-ment in mathematicsand the social envi-ronment of the school.

The frequency ofhome computer usewas positively relatedto academic achieve-ment and the socialenvironment ofthe school.

The use of computersto teach lower-orderthinking skills wasnegatively related toacademic achievementand the social envi-ronment of the school.

The frequency ofschool computer usewas unrelated to thesocial environmentof the school andnegatively related toacademic achievement.

For fourth-graders, thestudy found that:

Using computers forlearning games waspositively related toacademic achievement

3

and the social environ-ment of the school.

o By increasing thetendency of teachersto use computers forlearning games, profes-sional developmentof teachers was alsopositively related toacademic achievementand the social environ-ment of the school.

o The frequencies ofhome and schoolcomputer use werenegatively related toacademic achievementand the social environ-ment of the school.

The size of the relation-ships between the variouspositive uses of technologyand academic achievementwas negligible for fourth-graders but substantial foreighth-graders. For fourth-graders, professionaldevelopment and usingcomputers for learninggames each contributedabout a tenth of a gradelevel of academicachievement, or theequivalent of a fewweeks of instmction. Foreighth-graders, however,professional developmentand using computers forhigher-order thinking skillswere each associated withmore than a one-thirdof a grade level increase.

4

These findingscome with certaincaveats. First, the datawere collected at asingle point in time;the aspects of technol-ogy studied hereoccurred at the sametime as the educationaloutcomes of interest.Thus, it may be thathigh-achieving stu-dents are more likelyto use technology incertain ways ratherthan that these uses oftechnology promotehigh levels of aca-demic achievement.Second, while thestudy takes intoaccount some charac-teristics of teachers,it does not take intoaccount their overalltendency to teach incertain ways, such asto teach higher-orderthinking skills. It maybe that computers arebut one mediumamong many thatteachers use to teachhigher-order thinkingskills, and that allof these media areconducive to highlevels of academicachievement.

All of this suggeststhat computers areneither cure-alls forthe problems facingschools, nor mere fadsthat have no impact

on student learning.Rather, when they areproperly used, comput-ers may serve as impor-tant tools for improvingstudent proficiencyin mathematics, aswell as the overalllearning environmentin the school.

6

INTRODUCTION

In recent years, educa-tional technology hasbecome a cornerstonefor state and federalefforts to improve theperformance of thenation's school children."Educational technology"generally refers to theintroduction of comput-ers and related piecesof equipment to theclassroom. Across thenation, governors havetargeted a large share oftheir education budgetsto making computersmore available inschools. George Pataki,governor of New York,and Pete Wilson, gover-nor of California, haveasked their state legisla-tures for millions ofdollars to purchase newcomputers for schoolsand train teachers to usethem. Some states aretaking a top-downapproach, bringing thedecision-making skillsand computer expertiseof the state governmentto bear to move technol-ogy to the classroom asquickly and inexpen-sively as possible. WestVirginia, for example,purchased computers inbulk and sent them toschool districts; the stateis also responsible forteacher training. Otherstates are providing thefunds but allowingschool districts to makethe decisions. Kentucky,

for instance, requiredeach school district todevelop its own tech-nology plan and makeits own purchases ofsoftware and hardwarewith state funds (seeWhite, 1997).

State efforts toincrease access totechnology have beencomplemented byfederal action. In 1996,President Clinton andVice President Goreannounced the Tech-nology Literacy Chal-lenge. The overarchinggoal of the programwas to ensure that allstudents had access toeducational technology.Clinton and Goreidentified four pillarsof technological lit-eracy, namely moderncomputer hardware,the Internet, teacherdevelopment, and high-quality software andonline resources. TheTechnology LiteracyChallenge Grants havebeen awarded to statesto help them providethese elements. Thisand other funding fromthe Federal Govern-ment has resulted in afederal contribution of25 percent of all dollarsspent on educationaltechnology. To furtherassist in meeting thegoal of connectingstudents to theInternet, the Clinton

Administration initiatedthe E-Rate program,which requires thetelecommunicationsindustry to contributedollars to a fund tosupport school andlibrary Internet access(U.S. Department ofEducation, 1996).

Yet with all of thismoney being spent oneducational technology,many policymakers arewondering about theevidence on its effective-ness. The purpose ofproviding technologyto schools is to improvestudent academic perfor-mance and other educa-tional outcomes, not toprovide state-of-the-artequipment for its ownsake. What types ofinvestment in technol-ogy most improveacademic achievement?Does technology affectimportant educationaloutcomes at all? Unfortu-nately, for all of theinvestment in educa-tional technology, thereis a surprising lack ofhard data on its effects.State data collectionefforts are sketchy atbest, and the federalgovernment does notcollect national dataexpressly for thepurpose of evaluatingeducational technology.Prior research on com-puters in the classroomconsists primarily of

7

evaluations of exem-plary programs; thesestudies have significantmethodological prob-lems, and it is highlyquestionable whetherthe programs canbe replicated on alarge scale.

Fortunately, a greatdeal of information canbe gleaned from themost recent NationalAssessment of Educa-tional Progress (NAEP)for which data are nowavailable. NAEP consistsof tests in varioussubject areas adminis-tered to nationalsamples of students. Itis generally given everytwo years, and is usedto measure trends instudent performanceover time, as well asbetween subgroups ofstudents such as malesand females. NAEP alsoincludes questionnairesgiven to students, theirteachers, and theirprincipals, to put theNAEP test scores in theireducational context. The1996 NAEP, for the firsttime, included detailedquestions on technologyin its questionnaires. Byexamining these ques-tions and relating themto students' test scoresand other characteristicsof students, teachers,and schools, it ispossible to learn agreat deal about the

5

effectiveness of technol-ogy across the nation.

This report presentsthe results of a nationalstudy of technologicaleffectiveness using theNAEP in mathematics. Itfinds that technology canindeed be effective inincreasing mathematicsachievement and othereducational outcomes,but that this effectivenessdepends upon howtechnology is used.The report is organizedas follows:

o Chapter One presentsthe debate on techno-logy's effectiveness. Itthen discusses whatthe prior research sayson this issue, and findsit to suffer from meth-odological problemsthat limit its usefulnessto the debate.

o Chapter Two presentsdescriptive informationabout access to anduse of educationaltechnology by ournation's schoolchil-dren, particularly formathematical tasks.It finds that largegovernment invest-ments in technologyhave eliminated gapsin access to andfrequency of use ofcomputers, but thatlarge gaps in teacherpreparation andthe ways in which

6

technology is usedin the classroomremain.

o Chapter Threepresents the resultsof a statistical analy-sis of the relationshipbetween technology,academic achieve-ment in mathematics,and other educationaloutcomes. It findsthat the usefulness oftechnology dependsupon how the tech-nology is used; someuses are positivelyrelated to academicachievement andother educationaloutcomes, whileothers are not.

o Chapter Four dis-cusses the method-ological shortcom-ings of this studyand suggests someimplications of thestudy's findings forpolicymakers andpractitioners.

CHAPTER ONE:

THE DEBATE ON

TECHNOLOGY'S

EFFECTIVENESS

The penetration ofcomputers into the lifeof schools is a fact fewcould deny. The avail-ability of computersnationally has increaseddramatically in recentyears, from a computerfor every 125 studentsin 1983 to a computerfor every 9 students in1995 (Glennan &Melmed, 1996). Manyschools now have onecomputer for everytwo students. Yet theresponse of educatorsand practitioners hasbeen mixed. Some havebeen quite enthusiasticabout technology,seeing it as the toolmost needed to facilitatemore comprehensiveeducational reform.Others see it as apassing fad, more adistraction from schoolreform than anythingelse. Not all advocatesand critics subscribe tothe views paraphrasedbelow; these are simplyillustrations of a few ofthe prevailing viewsregarding technology.

THE CASE FOR TECHNOLOGY

Advocates of tech-nology argue for usingtechnology for thewholesale transformationof the classroom.' They

note that computers andsupporting technologieshave any number ofuses. These are generallyof five types: support forindividual learning,group learning, andinstructional manage-ment; communication;and administration.Applications to indi-vidual learning includedrilling students onparticular skills, usingCD-ROMs or the Internetto find resources notavailable in the school,communicating withexperts, word-process-ing, providing assistancein computations, anddemonstrating simula-tions of mathematicalor scientific concepts.Group learning applica-tions include usinge-mail to support groupcommunication, usingpresentation software toallow group presenta-tions on a project, andproviding collaborationin collecting and analyz-ing data. Applications toinstructional manage-ment include integratingstandards and assess-ments, managing studentportfolios, and develop-ing individual studentlearning plans. Commu-nications applicationsinclude communicatingto remote locations such

as rural schools andimproving communica-tion among students,teachers, and parents.Finally, applications toadministrative functionsinclude supportingattendance and account-ability functions.'Clearly, using technol-ogy in all of these wayswould result in thepenetration of technol-ogy into every aspectof the school day,theoretically makingconventional teachingtechniques obsolete.

This vision hasbeen applied, to somedegree, in variousinnovative schools.Blackstock Junior HighSchool in California, forinstance, developed"smart" classrooms, inwhich each student hasa computer, and theseare on a network. Theteacher can thus allowstudents to work eitherindependently or ingroups over the net-work. The school alsodeveloped the Tech Lab2000 to provide evenmore sophisticatedfacilities, such as Com-puter Assisted Designsoftware, a ComputerNumerically Controlledflexible manufacturingsystem, and even asatellite dish. Even

' The case of technology presented here draws heavily on Glennan & Melmed (1996). Also,see p_ge 9 for examples of studies supporting technology.

'An alternate schema for categorizing uses of technology can be found in Becker (1994a).

9 7

non-smart classroomsare technology-rich,with 10 computers andtwo printers per room.Other innovativeschools have similarlytechnology-rich layouts.Northbrook MiddleSchool in Texas hasone computer for everytwo students; eachclassroom has five tosix computers, com-plete with CD-ROMcapabilities, a school-wide network, andInternet access.

The primary indica-tor used by technologyadvocates to assesswhether a school isusing technology wellis how much hardwareand software hasactually reached theclassroom. Innovativeschools are identifiedbased upon theirstudent-to-computerratios, the sophistica-tion of their networksand peripherals, aswell as the amount ofmoney spent per-pupilon technology. In thisview, transformation ofeducational practicerequires a high level ofaccess to technology;without such access,technology is educa-tionally marginalized,rather than being an

essential feature ofinstruction. Thus a givenschool is not likely to beeffective in its uses oftechnology unless there isa substantial technologi-cal infrastructure.

In terms of the actualeffects of technology onacademic achievementand other educationaloutcomes, advocatesassert that most uses oftechnology are beneficial,and that they can lead toimprovements in most orall educational outcomes.They cite three pieces ofevidence. First, they notethat computers have beenused to provide studentswith opportunities fordrill and practice sincethe 1960s. This use,known as ComputerAssisted Instruction (CAI)has been evaluatedrepeatedly, and theevaluations suggest thatCAI students demonstratehigher levels of academicachievement than theircounterparts who do nothave CAI. Second, advo-cates note that morerecent uses of technol-ogy, such as those thatsupport higher-orderthinking skills, have notbeen as widely evaluated,but the few evaluationsdone, such as that ofApple Computers of

Tomorrow, showpromising results.Finally, they note thattechnology-rich schoolsseem to evince gains inachievement and othereducational outcomessuch as student motiva-tion and teacher morale.

THE CASE AGAINST

TECHNOLOGY

Others have beenquite critical of technol-ogy. They agree with theadvocates that the goalof spreading educationaltechnology is to use itto transform instruction;the critics, however,view this effort as beingdoomed to failure.3 Theirindicator for the successof technology programsis not the amount oftechnology available, butwhether teachers arewilling to use it. Nomatter how manycomputers are availablein the classroom, ifteachers are unwilling touse them for instruction,they are unlikely to havemuch impact on stu-dents. And, in the viewof the critics, teachersare by-and-large notmaking use of comput-ers. Schools tend to havea few teachers andadministrators who are

advocates of technologyand use it widely. Themajority of teachers,however, use technol-ogy as little as possible;they only do as muchwith computers asadministrators requireof them.

The critics do notclaim that teacherobstruction of technol-ogy is detrimental toschooling, but quite theopposite. They regardtechnology as unlikelyto improve academicachievement or indeedany other educationaloutcome of students. Insupport of this viewthey present threepieces of evidence: thehistory of technologicalinnovation, research onthe cost-effectivenessof CAI, and evidenceon theories of learning.First, they note thatteachers have histori-cally been resistant totechnological innova-tions when those inno-vations have made itmore difficult for themto get through thetypical school day. Earlytechnologies, such asblackboards and desks,were supported byteachers because theymade it easier forthem to manage the

3 The case against technology presented here draws heavily on Cuban (1986) and Cuban (1993). See pages 9 and 10 of thisreport for examples of studies criticizing technology.

8

classroom and conveyinformation. Laterinnovations, such asfilms and educationaltelevision were resistedbecause they wouldundermine the teacher-student relationship; tothe extent that thesetechnologies wouldsubstitute for teachers,teachers would haveless time to interactwith students. Indeed,the only situationsin which teachersembraced this technol-ogy were those inwhich teachers wantedto reduce interactionwith students; forinstance, when teacherswanted to waste timeduring low-energyperiods of the day(e.g. after lunch) theymight show a film.Computers, the criticsclaim, pose these sameissues and thereforewill be marginalizedor not used at allby teachers.

Second, researchon the cost-effective-ness of CAI shows thatwhile there may besome gains to academicachievement, these arenot proportionate tothe costs of buying and

maintaining computers;tutoring, for instance, wasfound to produce greatergains for less money.

Finally, cognitivetheories of educationsuggest that learningmay include a social,noncognitive element.Students learn not onlybecause they processinformation, but alsobecause of the complexreinforcements theyreceive from teachersand the socializationprocess in which learningis embedded. As comput-ers move from beingmere supplements tobeing the core of thelearning environment,they limit opportunitiesfor social interaction, thusinterfering with thelearning process. Hencecomputers should not beexpected to increasestudent learning and, infact, may decrease it.

THE EVIDENCE ON

TECHNOLOGY'S EFFECTIVENESS

There are someproblems with the asser-tions made by bothadvocates and criticsregarding the relationshipbetween technologyand educational

outcomes. With regardto the advocates, the CAIevaluations of drilland practice computerapplications, the evalua-tion of higher-orderapplications and thedata from the exemplaryprograms suffer fromcertain methodologicalproblems that raisequestions about theirfindings.4 First, they treattechnology as an undif-ferentiated characteristicof schools and class-rooms. No distinction ismade between differenttypes of technologyprogram, such as pro-grams that involveintensive professionaldevelopment or thosethat focus on higher-order thinking skills.Without differentiatingbetween programcomponents, it is notclear which areresponsible for overallprogram effectiveness; itmay be that some usesof technology are moreproductive than others.Second, these evalua-tions focus on a schoolor school district, butnever on statewide ornationwide uses oftechnology. It may bethat what works well in

Chicago will not workwell in New York City.Third, evaluations oftenuse a poor measure ofacademic achievement;many use tests that havebeen developed for thatparticular study andhave not been validated.Fourth, the students intechnology-rich environ-ments may be verydifferent from those inmore conventionalsettings. Many of thesestudies do not randomlyassign students todifferent environments,and some do not evenhave a group of stu-dents in a conventionalenvironment to whomto compare the studentsusing technology.Apparent gains inachievement may there-fore be attributable tothe student rather thanthe school. These prob-lems make the conclu-sion from these studies,that technology ispositively associatedwith academic achieve-ment, questionable.5

The evidence of thecritics also raises diffi-culties. The historicalargument that teachersobstruct technologybecause it is not useful

'For reviews of CAI, see Kulik & Kulik (1991), Niemiec et al. (1987), Liao (1992), and Christmann, Badgett & Lucking (1997).The reviews generally find significant differences in academic achievement and other outcomes between students exposed to CAIand those exposed to conventional teaching techniques, but find that these differences vary dramatically depending upon studymethodology, with some studies finding no difference or even a negative relationship.

'For an example of a critique of CAI research, see Colorado (1988).

9

assumes that computershave as little use foreducation as film stripsor educational television.To the extent that com-puters differ from thesetechnologies, theyshould be embracedby teachers. Indeed,research on teacherobstruction indicates thatthe inability of manyteachers to use comput-ers is based upon a lackof resources supportingteacher use of technol-ogy, not a Ludditefeeling among teachersthat technology isuseless. Henry J. Becker(1994b) analyzed datafrom one of the mostcomprehensive surveysof teacher technologyuse, the InternationalAssociation for theEvaluation of Educa-tional Achievementsurvey of 1989. Amongthe more than 1,000teachers surveyed,Becker did find both lowlevels of computer useand low levels of effec-tive use; only one of sixteachers used computersregularly in the class-room, and of these, amajority used themmostly for drill andpractice, avoiding morecomplicated applica-tions. (Just 16 percent of

10

mathematics teachersused computers forconveying "understand-ing of numerical rela-tionships" as opposed to58 percent using themfor "student masteryof computation.")Yet when he tried toaccount for differenttypes and levels ofcomputer use by teach-ers, Becker found thatteachers using comput-ers more effectivelywere more likely towork in schools offeringhigh levels of teacherdevelopment on com-puters and havingtechnology coordinatorsavailable to assistteachers with ongoingproblems. A morerecent study found thatthe primary reasonteachers were afraid oftechnology was a lackof experience usingcomputers (Rosen &Weil, 1995). Thusteacher obstruction isnot an insurmountablebarrier to computeruse; teachers simplyneed to be trained andsupported before theywill feel ready touse them.

The critics alsoargue that the CAIstudies show computersto be less cost-effective

than tutoring, and thattechnology-based educa-tion undermines thesocial, noncognitivecomponents of learning.6The problems with CAIstudies have alreadybeen discussed; regard-ing the social aspects oflearning, it can only besaid that the impact ofcomputers on learningdepends upon how theyare used. If they substi-tute entirely for humaninteraction, then theymay indeed underminesocial components ofinstruction. On theother hand, if theyare one important toolamong many, theywill still permit a greatdeal of face-to-facecontact between stu-dents and teachers.

At this point, then,the debate on tech-nology's effectivenessseems unresolved,leaving policymakerswondering about howto use technology or,indeed, whether to investin it at all. Research insupport of technologyseems to consist of small-scale studies with seriousmethodological problems;research against technol-ogy is based uponunproved assertionsthat computers are

no different fromfilm strips.

To some degree,the paucity of clear evi-dence on technology'seffectiveness stems fromdifficulties inherent in itsstudy. As Mandinachand Cline (in press)have noted, the lack ofcontrolled experimentsin this area reflects thefact that technology isnot introduced overa well-defined periodof time. Most experi-ments involve a specificintervention with a clearbeginning and end. Theintroduction of technol-ogy, however, involvesobtaining hardware andsoftware, training teach-ers, modifying curricula,and maintaining thetechnology, among otherthings. This process istherefore an ongoingone, without a clearstarting or stoppingpoint. Mandinach andCline suggest that evi-dence on technology'seffectiveness will haveto be drawn from variousnonexperimentalapproaches, usingmultiple methods atvarious levels (e.g. theclassroom, the school,the district).

6The study showing the differential cost-effectiveness of various interventions is reported in Levin, Desther & Meister (1986).

12

This report repre-sents one method ofstudying technologythat can, perhaps, helpbreak the statematebetween technology'sadvocates and critics.It presumes that thereality of technology'seffectiveness lies some-where between theoptimism of advocatesand the pessimism ofcritics. Some uses oftechnology are prob-ably conducive toacademic achievementand other positiveeducational outcomes,while other uses oftechnology are not.The report analyzesnational data on tech-nology that can distin-guish between practicesthat are effective andthose that are not, whilealso addressing someof the shortcomings ofprior research. Beforerelating patterns oftechnology use toacademic achievement,however, it will beuseful to examinethose patterns.

1311

CHAIPTER Two

PATTERNS OF

IEDUCATRONAL

TECHNOLOGY

USE

12

The purpose of thischapter is to describehow computers areused by students and tocompare usage betweendifferent types of stu-dents. Four key indica-tors of computer use arediscussed here:

o student access tocomputers in schoolfor mathematicaltasks;

o student access tocomputers andfrequency of com-puter use at home;

o preparedness ofmathematics teachersin computer use;

o the ways in which themathematics teachersand their students usecomputers.

In addition to knowingthe state of the nation inregard to these fourindicators, it is impor-tant to know how theyare distributed amongdifferent types of stu-dents. There may belarge differences incomputer access amongstudents, and thesedifferences may haveimportant policy impli-cations; if minoritystudents, for instance,have less access toteachers who havereceived training on

".4

computers, it might beimportant to increaseteacher training inschools with highproportions of minoritystudents. Six types ofsubgroups of studentsare particularly worthexamining, as theyappear frequently inpolicy discussions:

o Ethnicityo Gendero Region of the countryo Governance of school

(public vs. private)o Economic status (poor

vs. non-poor)o Community status

(urban, suburban,rural)

DATA AND MEASURES

The state of technol-ogy use can be mea-sured from data from theNational Assessment ofEducational Progress of1996 (NAEP). NAEP hasbeen administeredregularly since 1969 tonationally representativesamples of students ofvarious ages and gradelevels. The core of NAEPis an assessment that hasbeen given in varioussubject areas includingmathematics, science,reading, geography, andwriting. NAEP alsoincludes backgroundquestionnaires com-pleted by students, theprincipals in their

1 4

schools, and the teach-ers in the relevantsubject area. Studentsand teachers are askedabout their socialbackground, theirexperiences in schoolgenerally, and theirexperiences in theparticular subject area;principals are askedprimarily about schoolpolicies and practices.While earlier administra-tions of NAEP didinclude a few questionsabout technology, onlythe 1996 administrationsincluded sufficientinformation to measurethe four indicators listedabove. In 1996 NAEPwas administered tofourth-, eighth- andtwelfth-graders inmathematics andscience. This reportdescribes technologyuses among the 6,627fourth-graders and 7,146eighth-graders who tookthe core NAEP assess-ment in mathematics.

Student access tocomputers in school ismeasured by frequencyof computer use. Oftenresearchers use the ratioof computers to studentsor the level of spendingas an indicator of access.Yet computers may besitting in closets, or inclassrooms unused.Thus, to avoid questionsof whether schoolownership of computers

is the same as studentaccess, frequency of useby students is measuredinstead; use weekly ormore often is deemed tobe true access becauseless frequent use sug-gests that computers actmore as filler than aspart of the curriculum.For this study, studentreports of computer userather than teacherreports are used, becauseof how the relevantquestions are asked inNAEP. Students areasked: "When you domathematics at schoolhow often do you...use acomputer?" Teachers areasked "How often dostudents in this class...usea computer?" Studentsare thus responding withthe frequency with whichthey use computersspecifically for mathemat-ics. Teachers are respond-ing with the frequencywith which students intheir class are usingcomputers. For eighth-graders, the teacher mostlikely teaches mathemat-ics only, and so theresponses would reflectstudent computer use inmathematics only, but forfourth-graders theteacher probably teachesthe same students in avariety of subjects, and

so the responses wouldreflect computer usemore generally, thusoverstating mathemati-cally oriented use.Student responses aremeasured to avoid thisdifficulty. Studentaccess to a computer athome is also measuredby frequency of use.Whether students'families own computersis also included,because the NAEPquestion on frequencyof home use includesthe family not owninga computer as a choice.This information comesfrom the studentquestionnaire.7

Teacher prepared-ness is measured bywhether teachers havereceived professionaldevelopment in com-puter use. Presumably,teachers who receivetraining on computersare better prepared thanthose who do not; theBecker study reinforcesthis notion, finding thatteachers who receivetraining that is subject-specific or tool-orientedare indeed exemplaryusers of educationaltechnology. The alterna-tive is to use teachers'perceptions of theirpreparedness, which is

asked in NAEP. Yet theseperceptions of prepared-ness might not reflectreality; teachers couldthink they are preparedbut may not be. Also,changing such percep-tions is not as easy asfunding teacher training;the perception measurethus has less policyrelevance. The teacherdevelopment informa-tion comes fromteacher reports.

Finally, the ways inwhich computers areused is measured from aNAEP question thatsuggests four possibleuses. These are: "drilland practice," "demon-stration of new topics inmathematics," "playingmathematical/learninggames," and "simulationsand applications." Tocompare subgroups, theresponses that indicate ahigher-order thinkingactivity are placed inone category and thosethat indicate a lower-order thinking activityare placed in another.For fourth-graders, drilland practice is deemedthe lower-order activityand learning games thehigher-order activity.While many learninggames may simply bea form of drill and

7 Some research raises questions about the validity of even student reports (Becker, 1994c).

1 5

practice and thereforelower-order, otherlearning games maybe higher-order. It isunlikely that there willbe much opportunityfor the demonstrationof new topics orsimulations and appli-cations because somuch of fourth-grademathematics is compu-tational. Thus, only thelearning games cat-egory is likely toinclude substantialapplication of higher-order skills.

For eighth-graders,drill and practice areagain deemed lower-order, but simulationsand applications aredeemed higher-order.Because eighth-grademathematics doesintroduce algebraicand geometric con-cepts (among others),there is likely to beample opportunity toapply these concepts.Learning gamesprobably function asbusywork in theeighth-grade context,and so are neitherhigher-order norlower-order; newtopics are probablyhigher-order or lower-order dependingupon the topic, and

13

so are also notclassified either way(See Appendix foradditional informationon how this part of thestudy was conducted.)8

ACCESS TO SCHOOL

COMPUTERS

Frequency of com-puter use in schoolamong fourth-graders ispresented in Figure 1.Overall, 33.2 percent ofstudents reported usingcomputers in school formathematics work oncea week or more often.Access was somewhatgreater for Black stu-dents than for otherethnic groups; fully 41.9percent of Black stu-dents reported frequentuse (as opposed to 32.3percent of Hispanicstudents and 31.7percent of White stu-dents). There was alsosome regional differen-tiation; students in theSoutheast used comput-ers in schools morefrequently than studentsin the Midwest. Theother regions weresomewhere in themiddle. Finally, publicschool students usedcomputers more fre-quently than students

Figure 1: Percentage of Fourth-Graders Who Report UsingComputers at School at Least Once a Week

42

U.S. 33

Black

Asian 33132Hispanic 0

White 32

Male 34I 33Female

35School Lunch Eligible

32School Lunch Ineligible

37Southeast35Northeast

34West28Central

34Public

411, 28Private

Urban 35

Rural 33

32Suburban

i I I i I

0 10 20 30 40 50Percentage of Students

Note: Only values on opposite sides of the shaded area are statistically significantlydifferent from one another.

Source: 1996 NAEP Mathematics Assessment

8 The distinction between higher-order and lower-order thinking is of long standing. For the case that technology's virtues arerevealed primarily in higher-order thinking tasks, see Means (1994).

14

r3.

Figure 2: Percentage of Eighth-Graders Who Report UsingComputers at School at Least Once a Week

U.S. 28

Black 033Asian 30White 28

Hispanic 26

Male 31Female 26

School Lunch Ineligible 29School Lunch Eligible 026

Southeast 29Northeast 29

Central 28West 28

Private

Public 2834

Urban 31Suburban 28

Rural 25




1 7

attending private school.There were no differ-ences based upongender, economic status,or community status.

Eighth-gradersappear to use computersabout as frequently asfourth graders; 28.3percent reported usingcomputers once a weekor more often (Figure2). Unlike fourth-graders, however, therewere no regional differ-ences. Males had higherlevels of computer usethan females. Poorstudents had higherlevels of computer usethan non-poor students.And students attendingurban schools hadhigher levels of com-puter use than studentsattending mral schools.Also, among eighth-graders, private schoolstudents had higherlevels of computer usethan public schoolstudents.

These differencessuggest that traditionallydisadvantaged groupsdo not lag in access toschool computers.Compared to White,non-poor, and suburbanstudents, those who areminority, poor, andurban find at least asmuch opportunity to usecomputers in school.

15

ACCESS TO HOME

COMPUTERS

Overall, a majorityof fourth-graders haveaccess to a computer athome (58 percent, seeFigure 3). This accessvaries greatly betweengroups. Ethnic differ-ences were quite large.Among White and Asianstudents, two-thirds hadaccess to computers inthe home as opposed toless than half of Blackand Hispanic students.There were substantialregional differences.Less than half of thestudents attendingschools in the Southeasthad access to a homecomputer, as opposed totwo-thirds of students inthe Northeast; the otherregions were in themiddle. There were alsodifferences based uponschool governance,economic status, andcommunity status;private school studentswere more likely tohave home computeraccess (74 percent asopposed to 56 percentfor public school stu-dents); non-poor stu-dents were more likelyto have access (65.3percent as opposed to40.6 percent for poorstudents); and studentsattending suburban

16

Figure 3: Percentage of Fourth-Graders with Access to HomeComputers

U.S.

AsianWhite

Black

Hispanic

14544

58

68

64

Male 58

Female r 58

School Lunch Ineligible

School Lunch Eligible 41

65

Northeast =14, 65

Central r 61West l 57

Southeast po

Private

Public 56

74

Suburban 7. 63

01,5Urban 3iRural p2

I I i I I I




1 8

Figure 4: Percentage of Eighth-Graders with Access to HomeComputers

U.S. 064

White 069

Asian 68Black 051

Hispanic 0Male 066

Female 63

School Lunch Ineligible 72School Lunch Eligible 142

Northeast 068

West 066

Central 064

Southeast 060

Private

Public *6377

Suburban 69Urban 061

Rural 060

I I I I i I




/9

schools were morelikely to have access(63 percent as opposedto 52.7 percent and 51.7percent for urban andrural schools respec-tively). There were nogender differences.

Eighth-gradersshowed essentially thesame pattern (Figure 4).Overall, 64.3 percentreported a computer inthe home. Black andHispanic students weremuch less likely to haveaccess than White andAsian students. Studentsfrom the Northeast hadhigher levels of accessthan students from theSoutheast. Private schoolstudents had higherlevels of access thanpublic school students.Non-poor students hadhigher levels of accessthan poor students. Andsuburban students hadhigher levels of accessthan urban or ruralstudents. In the eighth-grade case, however,males had higher levelsof access than females.

A separate questionfrom how many stu-dents are in families thatown computers is howmany students usehome computers regu-larly if they have them.For fourth-graders,among those studentswho have access to a

17

home computer, 32.5percent used themweekly or more often,about the same as thelevel of school use(Figure 5). Differencesin frequency of use athome also mirroredthose of frequency ofuse at school. Blacksshowed higher levels ofuse than Whites andHispanics; 53 percent ofBlacks reported usingcomputers once a weekor more. There wassome regional differen-tiation, with Southeast-ern and Western studentsusing computers at homemore than Midwesternstudents. Among studentshaving computers in thehome, public schoolstudents and poor stu-dents also used themmore frequently thanprivate school or non-poor students. There wereno significant differencesbased upon gender orcommunity status.

Among eighth-graders with computersat home, the patternswere less interesting(Figure 6). In all, 46.6percent of eighth-gradersreported using computersat home weekly or moreoften; this is a higherlevel than for fourth-grade home use oreighth-grade school use.There were, however, nosignificant differencesbetween subgroups.

18

Figure 5: Percentage of Fourth-Graders Using the Computer atHome Once a Week or More

U.S. 33

BlackAsian

Hispanic

White

36313

29

Male 33lFemale 32

School Lunch Eligible

School Lunch Ineligible 29

1Southeast 38

41

West i 35Northeast 33

Central 26

Public

Private

33

27

53

Urban 711135Suburban 32

Rural Alll at_

1 I I 1 I 1 1

0 10 20 30 40 50 60Percentage of Students



Figure 6: Percentage of Eighth-Graders Using the Computer atHome Once a Week or More

U.S. 47

Asian 49White 047

Black 43Hispanic 43

Male 47Female 46

School Lunch Ineligible 046


West 150Central 0 47

Northeast 46Southeast 43

Private 551

Public 046

Urban 047Suburban 47

Rural 543

I I I i I I I

0 10 20 30 40 50 60

Percentage of Students



21

These patterns ofhome use suggest thatinequities exist in theavailability of computersat home, but not in theiruse. Poor, minority, andurban students are lesslikely to live in homesthat possess computersin both fourth andeighth grade. However,for those families thatown computers, studentuse tends to be as highor higher among disad-vantaged groups.

TEACHER PREPARATION

Among fourth-graders, four out of fivehad teachers whoreported receivingprofessional develop-ment in the use ofcomputers in the previ-ous five years (Figure7). These percentagesvaried by region of thecountry, school gover-nance, student eco-nomic status and com-munity status. Morepublic school teachersthan private schoolteachers reportedprofessional develop-ment, as did moreteachers of non-poorstudents than teachersof poor students, andmore suburbanteachers than urbanor rural teachers.

19

Of the regions of thecountry, the Southeasthad the most profes-sional development(85.6 percent of students'teachers) followed bythe West and the Mid-west, with the Northeasttrailing behind (with77.0 percent). Therewere no ethnic orgender differences.

For eighth grade(Figure 8), the percent-age of students withteachers reportingprofessional develop-ment was somewhatlower, but still quitehigh, at 76 percent.Again the Northeastlagged behind the otherregions of the country,and suburban schoolsoutstripped urbanschools on this charac-teristic. In this casehowever, male studentswere more likely tohave teachers who hadreceived professionaldevelopment than werefemale students. Therewere no significantdifferences basedupon ethnicity, schoolgovernance, oreconomic status.

These results suggestthat disadvantagedgroups, to some extent,receive less exposure toteachers well preparedin technology use thando other groups of

20

Figure 7: Percentage of Fourth-Graders Whose Teachers ReportArby Professional Development hi Technology Use in the PreviousFive Years

U.S. 81

White

Hispanic

Asian

Black

82818078

Male 82Female 81

School Lunch Ineligible F84I


Southeast n-1186

West

Central 9

Northeast OIA

Public

Private 164

Suburban

Urban

Rural

1183

I I I I

0 20 40 60Percentage of Students

11985.3

789751

80 100



22

Figure 8: Percentage of Eighth-Graders Whose Teachers ReportAny Professional Development in Technology Use in the PreviousFive Years

U.S. 76

White 7777Asian

76Black76Hispanic

Male 78Female 74

School Lunch Ineligible 7572School Lunch Eligible

West 8179Central

Southeast 77Northeast 65

Public 76Private 73

Suburban 78Rural 75

Urban 72

0 20 40 60

Percentage of Students80 100



23

students. Suburban andaffluent fourth-gradersare more likely to havetechnologically preparedteachers than urban andpoor fourth-graders;suburban eighth-gradersare more likely to havetechnologically preparedteachers than urbaneighth-graders, althoughthere are no differencesbased on economicstatus for this grade.

TYPES OF COMPUTER USE

The types of use towhich computers are putvaries greatly betweenfourth and eighth grade.Among fourth-graders,54.5 percent have teach-ers reporting learninggames as the primaryuse; 35.9 percent reportdrill and practice; 7.5percent report simula-tions and applications;and 2.1 percent reportintroducing new topics.Thus the activity tradi-tionally thought of asteaching higher-orderskillsapplying con-cepts or developingsimulations to illustratethemis rarely used.Given the computationalnature of fourth-grademathematics, however,it may be that whateverhigher-order thinkingis occurring is beingconveyed in learninggames. Among

21

eighth-graders, drill andpractice is the mostcommon activity, withteachers for 34.3 percentof students reporting it asthe primary use. Playinglearning games is thesecond most commonuse, followed closely byapplications (29.2 percentand 27.2 percent ofstudents respectively).Teachers report introduc-ing new topics as theprimary use just 9.2percent of the time. Thusit seems that for fourthand eighth grades,computers are often usedfor lower-order activities(i.e. drill and practice),but they are also fre-quently used for higher-order activities (learninggames for fourth-gradersand applications/simula-tions for eighth-graders).

There are, however,substantial variations inpatterns of computer usebetween subgroups.Among fourth-graders(Figure 9), there aredifferences based uponregion, school gover-nance, and communitystatus. Southeasternstudents are less likelyto play learning gamesand more likely to usecomputers for drill andpractice. And students inpublic and suburbanschools are less likely touse computers for suchgames, although there

22

Figure 9: Percentage of Fourth-Graders Whose Teachers ReportLearning Games and Drill as Primary Computer Uses

Drill

036Learning Games

U.S. 55

-057Asian

57Hispanic

White 56Black 48

56Female

Male ,54

School Lunch Ineligible 56School Lunch Eligible 55l

Northeast 6057West

54Central

Southeast 47

Private 61Public 0L54

Urban 60Rural 60

Suburban epi

035035042

--10 35038028---035036

44

39

03003803630 40 50 60 30 40 50 60




24

Figure 10: Percentage of Eighth-Graders Whose Teachers ReportSimulations/Applications and Drill/Practices as Primary ComputerUses

Simulations/Applications Drill/Practice

U.S. 027 34

Asian

White -----031Hispanic ---025

Black 014

Male 028Female 027

School Lunch Ineligible --033School Lunch Eligible 022

43

Central 46Northeast

We St 421Southeast 010

37

027

034

031

015

52

4445

Private 4130 010Public 427 36

Suburban 38Urban 018Rural Co

3734

r0 10 20 30 40 50 60 0 10 20 30 40 50 60




°

25

are no differences fordrill and practice. Thereare no significantdifferences basedupon ethnicity, gender,or economic status.

Among eighth-graders, there aredifferences in use forall categories exceptgender (Figure 10).Blacks are less likelyto use computers forapplications and morelikely to use them fordrill and practice. Usevaries with region aswell. The Southeast isthe most likely to drilland the least likely todo applications; theMidwest is the mostlikely to do applicationsand the least likely todrill. Also, suburban andnon-poor students aremore likely to usecomputers for applica-tions; public schoolstudents are more likelyto use computers fordrill and practice.

It appears, then,that there is little differ-ence in computer usebetween advantagedand disadvantagedgroups in fourth grade,but great differences ineighth grade. In eighthgrade, minority, poor,and urban students aremore likely to findthemselves learninglower-order skills thantheir White, non-poor,

23

and suburban counter-parts; disadvantagedstudents are alsoless likely to find them-selves learning higher-order skills.

In summary, someindicators of the organi-zation of computer usein schools raised equityissues while others didnot. Disadvantagedgroups do seem to beless likely to haveteachers well-preparedto use computers, anddisadvantaged eighth-graders seem to be lesslikely to be exposed tohigher order learningthrough computers.Disadvantaged studentsare also less likely tocome from families thatown computers. How-ever, there seems to beno disparity in access tocomputers at school, orin their frequency of useat home for thosefamilies that own one.It seems that policiesto promote computeraccess in school havesucceeded in eliminatinginequities of this sort;yet inequities in teacherpreparedness and whatis taught using comput-ers remain.

Identifying somedisparities does notsuggest whether thesedisparities matter. Theimportance of the

24

disparity for a givenindicator depends uponthe overall significanceof that indicator toacademic achievementand other educationaloutcomes. To determinethis, it is necessary tomeasure the relationshipsbetween each indicatorand these outcomes.

26

CHAPTER THREE :

MODELING

TECHNOLOGY' S

EFFECTIVENESS

As discussed in ChapterOne, the effectiveness ofeducational technologyhas traditionally beenmeasured throughtabulating the results ofevaluations of exemplarytechnology programs.While such evaluationsmay be useful in provid-ing examples of howtechnology can be suc-cessfully implemented,they suffer from certainshortcomings that make itdifficult to conclude thateducational technology is(or is not) related tovarious educationaloutcomes: The studies donot distinguish betweentypes of technology useor program components;they present a picture ofa particular classroom,school, or school districtrather than a nationwideor even statewide pic-ture; they often use poormeasures of educationaloutcomes, particularly ofacademic achievement;and they often disregarddifferences in the studentbodies between technol-ogy-rich and technology-poor environments.

How THE STUDY WAS

CONDUCTED

The current studytook a different approachto the measurement oftechnology's effective-ness. It conductedstatistical analyses of the

2 7

1996 National Assess-ment of EducationalProgress (NAEP) inmathematics for thetwo samples of studentspresented in ChapterTwo: 6,227 fourth-graders and 7,146 eighth-graders. Using thetechnique of structuralequation modeling, thestudy tested a model ofhow various technologycharacteristics might berelated to various educa-tional outcomes. Forinstance, the modelmight expect that profes-sional development ontechnology leads teach-ers to use computersto teach higher-orderthinking skills, which,in turn, improves testscores. In addition to sucha flow chart, the modeltakes into account howother nontechnologicalcharacteristics of schoolsand students might affectthe educational outcomes:The appearance of highertest scores in studentswho use technology morefrequently may be due tothe technology, or it maybe due to the fact thatsuch students come frommore affluent families,and so are better academi-cally prepared in the firstplace. Apparent higherachievement levels ofstudents with teacherswho are computer-proficient may be due tothis proficiency, or it

may be due to thesesame teachers havingmore teaching experi-ence and knowledge oftheir subject matter. Themodel is then testedagainst the data, in thiscase from NAEP, todiscover which relation-ships are significant,which relationships arenot, and how strongthe significant relation-ships are.

For this study, fourindicators of the organi-zation of technology useare related to two educa-tional outcomes and toone another. The fouraspects of technology are:

frequency of schoolcomputer use formathematical tasks;

access to home com-puters/frequency ofhome computer use;

professional develop-ment of mathematicsteachers in technologyuse;

higher-order andlower-order uses ofcomputers by math-ematics teachers andtheir students

Frequency of schoolcomputer use is basedon student reports andranges from "never orhardly ever" to "almostevery day." Frequency of

25

home computer use isthe same as schoolcomputer use, exceptthat students have theoption of reporting"there is no computer athome," which is consid-ered here the lowestpossible level of use.Professional develop-ment is based uponasking teachers whetherthey have received suchdevelopment in technol-ogy, particularly com-puters, in the previousfive years. And higher-order and lower-orderuses are measured foreighth-graders as "simu-lations and applications"for higher-order skillsand "drill and practice"for lower-order skills;for fourth-graders,higher-order thinkingis measured from"playing mathematical/learning games."

Two outcomes areconsidered: academicachievement in math-ematics and the socialenvironment of theschool. Academicachievement is mea-sured by scores on thecore mathematicsassessment. The socialenvironment is mea-sured by bringingtogether various indica-tors of how teachersand students feel

26

about going to school.These are:

o student tardiness;o student absenteeism;o teacher absenteeism;o teacher morale;o student regard for

school property

All of these indicatorscome from reports byprincipals.

In addition, threecharacteristics of stu-dents and schools aretaken into account in themodel. These are:

o student socioeco-nomic status (SES);

o class size;o teacher characteristics

This means that allrelationships betweentechnology and educa-tional outcomes reportedhere represent the valueadded by technology forcomparable groups ofstudents with compa-rable teachers in compa-rably sized classes.

For both fourth- andeighth-graders, therelative affluence of astudent and the student'sfamily is measured froma series of indicators,including parent educa-tion levels and posses-sions in the home.9 Alsofor both fourth- and

eighth-graders, class sizeis measured from ateacher report of howmany students are intheir class. For fourth-graders, the teachercharacteristic measuredis teacher education level(less than B.A., a B.A.,an M.A., a Ph.D.). Foreighth-graders, teachereducation levels arecombined with years ofexperience and whetheror not the teacher hadmajored or minored inmathematics or math-ematics education.

This approachshould address theshortcomings of earlierresearch mentionedabove. It distinguishesbetween differentaspects of the organiza-tion of technology use;it is national in scope,analyzing nationalsamples of students; ituses robust and well-validated measures ofacademic achievement,as well as presentingsome noncognitiveoutcomes; and it takesinto account differencesin student bodies, aswell as school character-istics. (For more infor-mation on how thestudy was conducted,see Appendix.)

MODEL RESULTS: FOURTH

GRADE

As seen in Figures11 and 12, many of thetechnology characteris-tics are related to oneanother. School com-puter use is positivelyrelated both to homecomputer use andteacher professionaldevelopment. Thismeans that studentswho use computers inschool with great fre-quency are more likelyto use them at homeand more likely to haveteachers who havereceived professionaldevelopment in technol-ogy. Professional devel-opment in technologyappears to be related tostudents' playing learn-ing games; this meansthat teachers who havehad professional devel-opment on computersare more likely to usethem in that way.

These technologycharacteristics appear tohave different relation-ships to the two out-comes. In both cases, therelationship to learninggames seems positive,while the relationships tofrequency of school andhome computer use arenegative. Professionaldevelopment is also

9 Some questions have been raised regarding the validity of student reports of these measures (Berends & Koretz, 1995).

2 8

Figure 11: Influences on Academic Achievement Fourth-Graders

Relationships take into account student socioeconomic status, teacher characteristics, and average class size.

Figure 12: Influences on Social Environment Fourth-Graders

ELearningGames


2927

related to both outcomes,but indirectly; by virtueof increasing the useof learning games, itincreases academicachievement and improvesthe social environment.

MODEL RESULTS: EIGHTH

GRADE

For eighth-graders,the types of technologyare again related to oneanother. School use ispositively related to homeuse, indicating thatstudents who use com-puters frequently inschool are also morelikely to use them

frequently at home.Professional develop-ment is positivelyrelated to higher-orderthinking, suggestingthat teachers who areknowledgeable in theuse of computers aremore likely to usethem for higher-orderpurposes.

Again, these tech-nology characteristicsappear to have differ-ent relationships toacademic achievementand social environment(Figures 13 and 14).School computer useis negatively relatedto mathematics

achievement although,in this case, homecomputer use is posi-tively related to math-ematics achievement.Thus, students who usecomputers frequently athome demonstratehigher levels of aca-demic achievement,whereas those who usecomputers frequently atschool evince lowerlevels. Yet since schoolcomputer use is associ-ated with home com-puter use, school com-puter use does not havewholly negative conse-quences for academicachievement; students

who use computers atschool frequently arealso more likely to usethem at home fre-quently, and the latteruse is positively relatedto academic achieve-ment. The net effect ofschool computer use isnegative, but it is miti-gated by this indirectrelationship. Profes-sional development andhigher-order thinkingare both positivelyrelated to academicachievement: Studentswith teachers who havehad such professionaldevelopment showhigher levels of

Figure 13: influences on Academic Achievement Eighth-Graders


28 I30

achievement, as dothose who are taughthigher-order skills withcomputers. Professionaldevelopment is alsoassociated with aca-demic achievementindirectly; it increasesthe likelihood of teach-ers using computers toconvey higher-orderskills, thereby increas-ing academic achieve-ment in that way.Finally, using computersfor drill and practice,the lower-order skills, isnegatively related toacademic achievement.For social environment,the pattern is the same,

except that schoolcomputer use is notdirectly related to it; itonly has the indirectbenefit of increasinghome use, which isconducive to the socialenvironment in school.

THE SIZE OF THE

RELATIONSHIPS

While the flowcharts say a great dealabout the differentialeffectiveness of thetechnology indicators,they do not quantifythat effectiveness. Suchquantification can beaccomplished in two

ways. First, the size ofthe relationships can bemeasured in terms ofestimated grade levels.This is accomplished bytaking the differences inNAEP scores betweenthe characteristics anddividing by 12.25, thetypical difference inmathematics achievementbetween grade levels. Yetthis says little about thesize of the relationshipsrelative to other charac-teristics of schools andstudents, such as varia-tions in class size. Fortu-nately, the technologycharacteristics can becompared to other

characteristics in themodel by standardizingthem, meaning that theyare all placed on thesame scale; the numbersthat result are meaning-less in absolute terms,but do show the strengthof each relationshiprelative to the others. Inboth cases, it should beemphasized, thesenumbers represent thevalue added by technol-ogy above and beyondstudent, teacher, andclassroom characteristics.

The grade-levelestimates suggest sub-stantial positive benefitsof technology for

Figure 14: Influences on Social Environment Eighth-Graders


29

eighth-graders, but mixedresults for fourth-graders(Table 1). In fourth grade,students using computers toplay learning games are .15of a grade level ahead oftheir counterparts; studentswith teachers who havehad professional develop-ment on computers are .09of a grade level ahead.Assuming a 36-week schoolyear, these gains amount tothree and five weeksrespectively. In eighthgrade, students usingcomputers for higher-orderthinking skills show gainsof .42 of a grade level, andthose with a teacher whohas received professionaldevelopment on computersshow gains of .35 of agrade level. These wouldamount to much moresubstantial gains of 15 and13 weeks respectively.

The size of the technol-ogy relationship relative toother characteristics alsoemphasizes differencesbetween fourth- and eighth-graders (Table 2).1° SES hasthe largest relationship toachievement, at .58 forfourth grade and .40 foreighth grade. The sums ofthe class size and teachercharacteristic relationshipsare much smaller, totaling.08 and .13 for fourth- andeighth-graders respectively.

Table 1- Size of Relationships: Grade Levels

Variable Fourth Grade Eighth Grade

ProfessionalDevelopment .09 .35

Primary UseApplications n/a .42

Primary UseGames .15 n/a

Primary UseDrill n/a -.59

School ComputerUse -.20 -.11

Home ComputerUse -.26 .14

Table 2- Size of Relationships: Relative Sizes

Variable Fourth Grade Eighth Grade

SocioeconomicStatus .58 .40

Class Size .07 .06

TeacherCharacteristics .01 .07

ProfessionalDeveloment .01 .04

Primary UseApplications n/a .04

Primary Use -Games .03 n/a

Primary Use -Drill n/a -.06

School ComputerUse -.09 -.05

Home ComputerUse

-.14 .07

'° The positive relationship between class size and mathematics achievement indicates that smaller classes are associated with higher levels ofachievement.

30 3 2

The benefits of technol-ogy are smaller still, at.04 for professionaldevelopment and learn-ing games in fourthgrade, and .08 for profes-sional development andhigher-order thinking ineighth grade. Yet whilethe impact of technologyis smaller in both cases,it plays a larger role ineighth grade than infourth grade; in eighthgrade it is more than halfas great as class size andteacher characteristicsand a fifth as large asSES; for fourth grade it isone-half the size of classsize and teacher charac-teristics, but one-four-teenth as large as SES.

There were alsodifferences in relation-ship sizes for those typesof technology that arenegatively associatedwith academic achieve-ment. For fourth-graders,these negative relation-ships are larger thanthe positive; while thepositive total .04, thenegative total .23. Onthe other hand, for theeighth-graders thepositive relationshipsoutweigh the negative; ifusing home computers is

included, the positiverelationships total .15while the negativerelationships total .11.

These results, then,paint a more completepicture of technology'seffectiveness. In general,the indicators of profes-sional development andhigher-order uses ofcomputers seem posi-tively related to academicachievement in math-ematics and other out-comes, while frequencyof use is unrelated oreven negatively relatedto them. Further, whilethe size of the positiverelationships are substan-tial for eighth-graders,they are small for fourth-graders, so small thatthey are overshadowedby various negativetechnology relationships.The following chaptersuggests what can bemade of these findingsfrom a policy perspective.

3331

CHAPTER TouRg INTERPRETING THE

RESULTS

CONCLUSIONThe results from

this study suggest that,as technology advo-cates have asserted,technology does matterto academic achieve-ment, with the impor-tant caveat that whetherit matters dependsupon how it is used.The levels of use ofcomputers seems not tomatter, and extremelyhigh levels of use mayeven be counterproduc-tive. Possibly at suchhigh levels students areusing computers inunproductive ways,such as playing non-educational games. Butwhen computers areused to perform certaintasks, namely applyinghigher order concepts,and when teachers areproficient enough incomputer use to directstudents toward pro-ductive uses moregenerally, computers doseem to be associatedwith significant gains inmathematics achieve-ment, as well as animproved social envi-ronment in the school.

The findings regard-ing home computersreinforce this point.Students using homecomputers frequentlyhad higher levels of

32

achievement in eighthgrade, but lower levelsof achievement in fourthgrade. Presumably thiswas so because thecomputer was put todifferent uses in the twogrades. Perhaps eighth-graders have moreunderstanding of how touse the Internet thanfourth-graders, and socan use it to providesupplementary informa-tion on academic sub-jects; perhaps eighth-graders are more likelyto use computers astools for doing home-work, including usingword processors forwriting papers andspreadsheets for doingcalculations and tabulat-ing data. Without havingactual data on howhome computers areused by students, it isonly possible to specu-late, but perhaps ifhome computers areacademically productivefor eighth-gradersand not for fourth-graders they are beingused in substantiallydifferent ways.

The descriptiveinformation in ChapterTwo, when combinedwith the results of theanalyses in ChapterThree, suggests theimportance of variousinequities in technologybetween groups ofstudents. Disadvantaged

3 4

groups seem to lagbehind in access tothose aspects oftechnology that doaffect educationaloutcomes, but not inaccess to those aspectsof technology that donot affect educationaloutcomes. While minor-ity, poor, and urbanstudents are no lesslikely to use computersat school frequently,frequency of use is notassociated with gains inachievement or socialenvironment. Yet minor-ity, poor, and urbanstudents are less likelyto receive exposure tocomputers for higher-order learning, and poorand urban students areless likely to haveteachers who havereceived professionaldevelopment on technol-ogy use. Thus, wheretechnology matters, thereare significant inequities;only where technologydoes not matter havethese inequities beensuccessfully erased.

KNOWING MORE

While this studydoes address problemsin the prior literature, ithas its own method-ological limitations.First, the data used werecollected at a singlepoint in time; technol-ogy characteristics occur

at the same time as theeducational outcomes.There are no priormeasures of mathematicsachievement, making itdifficult to rule out thepossibility that positiveeducational outcomesare conducive to certainaspects of technologyuse rather than the otherway around. The studydid address this problemin part by taking intoaccount student back-ground, a proxy forprior achievement.

Also, an alternatemodel for eighth-graderswas developed that tookinto account the level ofdifficulty of the eighth-grade mathematics class(algebra, pre-algebra, orregular eighth-grademathematics). Since theclass that students takedepends to some degreeon prior achievement,this measure may alsoserve as a proxy for it.The results from thealternate model, how-ever, did not differ muchfrom those of the originalmodel. All technologymeasures remainedsignificantly related toacademic achievementand the social environ-ment of the school, andthe sizes and directionsof the relationships alsoremained the same. Theone exception wasprofessional develop-ment, which, while still

significantly and posi-tively associated with theeducational outcomes,was associated with themto a somewhat lesserextent, with a relation-ship size of .21 gradelevels (as opposed to.35 grade levels). Therelative lack of changebetween the two modelssuggests that a study thattook prior achievementinto account would beunlikely to produce verydifferent results. None-theless, a study thatfollows students overtime, measuring theiracademic achievementboth before and aftertheir exposure to varioususes of technologywould provide a strongertest of the findings fromthis study.

Second, while thestudy does take intoaccount teacher charac-teristics, such as educa-tion levels, it does nottake into account moredetailed measures ofteacher practices. Thestudy found that whencomputers were used forhigher-order thinkingskills, students per-formed better; yet thisdoes not exclude thepossibility that studentsdo better when teacherstend to teach higher-order thinking skills,regardless of themedium. Fortunately,this invitation does not

33

undermine the findingsof this study. Regardlessof whether other mediamay also be successful atconveying higher-orderthinking, this study doesdemonstrate that com-puters can do so; theyare, at a minimum, oneof a limited number oftools that can contributeto this goal.

A few other caveatsare worth noting. Thestudy only analyzestechnology's effective-ness in one subject area,mathematics. Mostmeasures of the uses oftechnology as well as themeasure of academicachievement are specificto that subject. Comput-ers may be less effectivein other subject areas,such as history andEnglish. Also, the studyis limited to only a fewindicators of the organi-zation of technology use;other aspects of technol-ogy may be more or lesseffective. For instance,the study does notdistinguish between theeffectiveness of differenttypes of software.

Finally, the studydoes not incorporateinformation about statetechnology policies,which may have abearing on technology'seffectiveness. Thesethree caveats all suggestthat it is important forstates to collect more

data to evaluate theirtechnology investments;they need to knowwhich policies and typesof software are effective,and for which subjectareas computers shouldbe used.

IMPLICATIONS OF FINDINGS

These methodologi-cal limitations aside, thefindings have significantimplications for technol-ogy policy and practice.

First, the studysuggests that federaland state policymakersshould redouble theirefforts to ensure thatteachers are properlytrained to use computers.The descriptive informa-tion supplied in ChapterTwo shows that thevast majority of teachersalready receive someprofessional developmentin technology, suggestingthat there is little roomfor policymakers tofurther increase suchprofessional develop-ment. Yet there are largedisparities in the amountof professional develop-ment teachers receive.Federal and state policy-makers should targetteacher training efforts athigh-poverty urban andrural schools, which havemuch lower percentagesof teachers receiving suchtraining. Also, profes-sional development in

33

technology, as reportedin NAEP, could refer toanything from a week-end seminar to a semes-ter-long course. Manyof the teachers who doreceive professionaldevelopment in technol-ogy may benefit frommore intensive instruc-tion. Federal and statepolicymakers shouldmake sure that thequality of the teachertraining offered is highand intensive, sincethis training is such animportant componentof making technologyuse successful.

Second, the studysuggests that teachersshould focus on usingcomputers to applyhigher-order skillslearned elsewhere inclass. Computers shouldbe a component ofa seamless web ofinstruction that includesnontechnological com-ponents. For example,teachers might introducenew topics and conveybasic information totheir students throughgeneral class discussionand lecture, then assignprojects and problemsthat computers as wellas other media (books,field trips, etc.) can beused to address. Whileimplementing this visiondepends primarily uponthe individual prefer-ences and capabilities

34

of teachers, federal andstate policymakers canencourage it to somedegree through sup-porting professionaldevelopment of teach-ers that emphasizesthis teaching method.Further, federal andstate policymakers canencourage schooldistricts to order soft-ware that draws onthese skills, and assistschool districts inconnecting to theInternet, which can beused to obtain outsideinformation for projectsand problem solving.As with teacher devel-opment, extra effortsshould be made tosupport these changesin high-poverty urbanand rural school dis-tricts, which are atpresent most likely tobe using computers fordrill and practice ratherthan applications andsimulations. This is notto say that the basicsshould not be taughtwhere appropriate; thefindings merely suggestthat computers are notwell-adapted to teach-ing them.

Third, the primaryfocus of all technologyinitiatives should beon middle schoolsrather than elementaryschools. The effects oftechnology appear tobe much smaller in the

4'1

fourth than the eighthgrade, and so may notbe cost-effective. Fur-ther, the sequence of thetypical mathematicscurriculum suggests thatcomputers are morecrucial for middle schoolstudents than elementaryschool students. Mosthigher-order conceptsare not introducedbefore middle school,with elementary schoolstudents focusing oncomputational skills.To the extent that theprimary benefit ofcomputers lies in apply-ing higher order skills,there may not be muchopportunity to benefitfrom using computersbefore middle school.

In conclusion,technology advocateswere correct in assertingthat technology can bebeneficial to studentlearning. Used properly,technology can leadto gains in academicachievement and posi-tively influence thesocial environment ofthe school, reducingteacher and studentabsenteeism and increas-ing morale. Yet it isimportant that the scopeof technology in schoolsbe limited to those areaswhere it provides ben-efits, and reduced inareas where it doesnot. Thus the notionof technology as a

36

substitute for conven-tional forms of instruc-tion, often hoped for bytechnology advocatesand dreaded by technol-ogy critics, may over-state the case for tech-nology use. It may alsobe counterproductive,not only introducingcomputers into areaswhere other teachingtechniques are moreconducive to positiveeducational outcomes,but also raising worriesamong teachers thatthey are being replacedby machines. By clearlydelineating areas inwhich computers canbe helpful to teachersand areas in whichthey cannot, it will bepossible to increase theacceptance of comput-ers. Alongside chalkand blackboards, com-puters will be toolsteachers feel theycannot live without.

REFERENCES Arbuckle, J. L. (1997).AMOS users' guide:Version 3.6. Chicago:SmallWaters Corpora-tion.

Becker, H. J. (1994a).Making sense of newtechnologies. Execu-tive Educator, 16,12-13.

Becker, H. J. (1994b).How exemplarycomputer-usingteachers differ fromother teachers: Impli-cations for realizingthe potential of com-puters in schools.Journal of Research onComputing in Educa-tion, 26, 291-320.

Becker, H. J. (1994c).Validity of studentreports in NELS:88:The case of high-endcomputer use inscience classes. Annualmeeting of the Ameri-can EducationalResearch Association,New Orleans.

Berends, M. & Koretz, D.M. (1995). Reportingminority students' testscores: How well canthe National Assess-ment of EducationalProgress account fordifferences in socialcontext?, EducationalAssessment, 3,249-285.

Christmann, E., Badgett,J., & Lucking, R.(1997). Progressivecomparison of theeffects of computer-assisted instruction onthe academic achieve-ment of secondarystudents. Journal ofResearch on Comput-ing in Education, 29,325-336.

Colorado, R. J., (1988).Computer-assistedinstruction research: Acritical assessment.Journal of Research onComputing in Educa-tion, 20, 226-233.

Cuban, L. (1986). Teach-ers and Machines: 7heclassroom use oftechnology since 1920.New York: Teacher'sCollege Press.

Cuban, L. (1993). Com-puters meet classroom:Classroom wins.Teacher's CollegeRecord, 95, 185-210.

Glennan, T. K., &Melmed, A. (1996).Fostering the use ofeducational technol-ogy: Elements of anational strategy.Santa Monica, CA:RAND.

Gustafsson, J.E., & Stahl,P.A. (1997). S7REAMSusers' guide: Version

37

1.7. Molndal, Sweden:Multivariate Ware.

Johnson, E.G. (1989).Considerations andtechniques for theanalysis of NAEP data.Journal of EducationalStatistics, 14, 303-334.

Johnson, E.G., Mislevy,R.J., & Thomas, N.(1994). Scaling proce-dures. In E.G. Johnsonand J.E. Carlson (Eds.),The NAEP 1992Technical Report (pp.241-256). Princeton,NJ: Educational Test-ing Service.

Kulik, C. C., & Kulik, J.A. (1991). Effective-ness of computer-based instruction: Anupdated analysis.Computers in HumanBehavior, 7, 75-94.

Levin, H., Desther, D., &Meister, G. (1986).Cost-effectiveness ofalternative approachesto computer-assistedinstruction (IFGReport). Stanford, CA:Stanford University.

Liao, Y. K. (1992). Effectsof computer-assistedinstruction on cogni-tive outcomes: A meta-analysis. Journal ofResearch on Comput-ing in Education, 24,367-380.

35

Mandinach, E., & Cline,H. F. (in press). Itwon't happen soon:Practical, curriculum,and methodologicalproblems in imple-menting technology-based constructivistapproaches in class-rooms. S. P. Lajoie(Ed.), Computers ascognitive tools: Thenext generation.Mahwah, NJ: LawrenceErlbaum Associates.

Means, B. (1994). Intro-duction: Using tech-nology to advanceeducational goals. InB. Means (Ed.), Tech-nology and educationreform: The realitybehind the promise.San Francisco: Jossey-Bass Publishers.

Niemiec, R., Samson, G.,Weinstein, T., &Walberg, H. (1987).The effects of com-puter-based instructionin elementary schools:A quantitative synthe-sis. Journal ofResearch on Comput-ing in Education, 19,85-103.

Quality Education Data,Inc. (1994). Technol-ogy in public schools,QED's 13th annualstudy of public schooltechnology use. Den-ver, CO: Author.

36

Rosen, L.D., & Weil, M.(1995). Computeravailability, computerexperience andtechnophobia amongpublic school teach-ers. Computers andHuman Behavior, 11,9-31.

U.S. Department ofEducation. (1996).Getting America'sstudents ready for the21st Century: Meetingthe TechnologyLiteracy Challenge.Washington, D.C.:U.S. GovernmentPrinting Office.

White, K. (1997). Amatter of policy.Technology Counts:Schools and Reformin the InformationAge, 17, 40-42.Washington, D.C.:Editorial Projects inEducation.

38

APPENDIX:

How THESTUDY WAS

CONDUCTED

CASE AND VARIABLE SELECTION

Data were drawnfrom the 1996 NationalAssessment of EducationalProgress in mathematicsfor fourth- and eighth-graders. For each grade,student and teacher datawere linked to school datausing the school identifier,SCRPSU. Only those caseswhich had plausible valuesfor the core assessments,MRPCM1, MRPCM2,MRPCM3, MRPCM4,and MRPCM5, were used,resulting in 6,227 fourth-graders and 7,146eighth-graders.

To measure socio-economic status the follow-ing six variables wereselected: B003501M(mother's education),B003601M (father's educa-tion), B000901M (newspa-pers), B000903M (encyclo-pedias), B000904M (books),and B000905M (magazines).

To measure teachercharacteristics, the follow-ing six variables wereselected: T040301 (yearstaught), T056301 (highestdegree), T040703 (math-ematics major), T040704(mathematics educationmajor), T040803 (math-ematics minor), andT040804 (mathematicseducation minor). Classsize was measured fromT044000. Academicachievement wasmeasured from the five

39

plausible value variablesused for case selection.

To measure socialenvironment, the follow-ing five variables wereselected: C032402 (stu-dent absenteeism),C032401 (student tardi-ness), C032406 (teacherabsenteeism), C032502(teacher morale), andC032506 (regard forschool property).

With regard to thetechnology variables,frequency of schoolcomputer use was takenfrom M812710B; fre-quency of home com-puter use was takenfrom B009301M; teacherdevelopment was takenfrom T056702; andprimary use was takenfrom T057601. All vari-ables were recoded toeliminate missing values.

Whether teachershad received educationin mathematics, one ofthe teacher characteris-tics, was calculated bytreating a "yes" responseto any of T040703,T040704, T040803, orT040804 as havingreceived such education.

The two frequency-of-computer-use vari-ables were recoded sothat a high value de-noted high frequency.The class-size variablewas recoded so that asmall class had a highvalue, and the social

environment measureswere recoded so thata high value indicateda positive socialenvironment.

DESCRIPTIVE ANALYSES

All cases wereweighted by the variableORIGWT divided by themean weight for eachsample. The overallnumber of cases wasalso divided by a designeffect of 1.75. Compari-sons of means were thenconducted using theScheffe test.

MULTIVARIATE ANALYSES

Models of the rela-tionship between tech-nology characteristicsand educationaloutcomes were devel-oped based upon theliterature. The modelswere tested for fourth-and eighth-gradersseparately using thetechnique of StructuralEquation Modeling asimplemented by twosoftware packages:STREAMS 2.0 and AMOS3.6; STREAMS is a pre-and postprocessor forAMOS. Unidimensionalmeasurement modelswere developed foracademic achievement,social environment, SES,and teacher characteris-tics based upon their

37

corresponding variablesin the database. Forthe remaining variables,the constructs weredefined as the databasevariables with a factorloading of 1 and nomeasurement error.

The constructs wererelated to one another asfollows: for fourth-graders, the technologyindicators, class size,teacher characteristics,and SES were all relatedto academic achievementand school social envi-ronment. Professionaldevelopment was relatedto use of learning games.Social environment wasrelated to academicachievement. SES wasrelated to frequency ofschool computer use andboth of those wererelated to home com-puter use. Frequency ofschool computer use wasrelated to professionaldevelopment. For eighth-graders the model wasthe same, except thatfrequency of schoolcomputer use was notrelated to professionaldevelopment and profes-sional development wasrelated to simulation/applications and drill andpractice rather thanlearning games. Modelswere deemed acceptablewith goodness of fitindices better than .9and a root mean squared

38

error of approximationbetter than .05. Coeffi-cients were deemedsignificant with stan-dard errors inflated bythe square root of thedesign effect (1.75) atthe .05 level. Sensitivityanalyses were con-ducted in which stan-dard errors were furtherinflated by the measure-ment variability of theplausible values, andthe measurement errorin the measurementmodel was correspond-ingly eliminated; thisprocedure producedsimilar results. ForSTREAMS 2.0, seeGustafsson and Stahl(1997). For AMOS 3.6see Arbuckle (1997). Fordiscussions of weight-ing, design effects, andmeasurement variabilityusing NAEP, seeJohnson (1989), andJohnson, Mislevy, &Thomas (1994).

4 0

04202-15785 5911M7 204115BEST COPY AVAILABLE

fa

D

U.S. Department of EducationOffice of Educational Research and Improvement (0ERI)

National Library of Education (NLE)Educational Resources Information Center (ERIC)

NOTICE

REPRODUCTION BASIS

®

ERIC

This document is covered by a signed "Reproduction Release(Blanket) form (on file within the ERIC system), encompassing allor classes of documents from its source organization and, therefore,does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission toreproduce, or is otherwise in the public domain and, therefore, maybe reproduced by ERIC without a signed Reproduction Release form(either "Specific Document" or "Blanket").

EFF-089 (9/97)

document resume ed 425 191 · 2014-05-19 · document resume. tm 029 255. wenglinsky, harold does...

Documents