Original

Educ. Technol. Res., 9, 13-30, 1986

Educational Technology in the Japanese SchoolsA Meta-Analysis of Findings

Barbara J. SHwALB,*1,

* 3 David W. SHwALB*1

> * 4 and Hiroshi AZUMA* 2

*'Center for Research on Learning and Teaching, University of Michigan,109 E. Madison, Ann Arbor, MI, 48109 U.S.A.

# 2Faculty of Education, The University of Tokyo,7-3-1 Hongo, Bunkyo-ku, Tokyo, 113 Japan

Received for publication, June 25, 1985

I. BACKGROUND OF THE PROBLEM

employ technology?," and "Are teacher-studentrelationships weakened when instruction by

In Japan, as elsewhere, education is a growing, technology is used?" However, such statements

massive enterprise. Currently more than 22 can be evaluated.million children are enrolled in Japanese pri- In Japan and in the U.S. many individual

mary and secondary schools. They are taught by research projects have compared the effectiveness

over one million teachers and supported by of traditional lecture/discussion instruction to

more than 10% of the national budget. Within varied forms of technology instruction. Whether

such a complex enterprise, innovations come or not we support the merit of such a limited

slowly. But they do come, and change has definition, effectiveness is almost always a

always characterized Japanese education. Most measure of student learning. And with regard to

recently change has been brought about by student learning, the reports are inconsistent.

technology uses in educational procedures.

Some reports showed technology to (1) result in

Such change has its enemies and friends. less student learning; (2) to produce greater

Opponents of technology say machines cannot student learning; and (3) to make no difference

teach as well as trained human beings, technology in student learning. Piecemealing together an

separates students from books resulting in im- answer to the question, "How well does tech-

personalized, dehumanizing teaching, and tech- nology teach?" from separate reports would be

nology increases distance between pupils and similar to trying to grasp the sense of hundreds

teachers. Proponents counter by saying that of test scores without using statistical methods

although teachers are deeply concerned, dedi- to organize, depict and interpret the data.

cated people they are subject to human frail- However, these individual research projects do

ties of insufficient skills, and attitudes and pre- provide a data based for more rigid evaluations.

judices reflecting limited background and ex-

A somewhat more rigid evaluation of research

perience. Moreover, they experience strong results can be made using the "review of the

pressures to guide instruction toward entrance literature" method. Reviews of this method

examinations and from special interests of summarize individual research projects and are

teachers' unions and parents' groups. Proponents of two major types: narrative and box-score.

believe technology can be more responsive to Narrative reviews seldom win converts to op-

individual student needs than can books and posing viewpoints because they are subjective

that technology distances a teacher from a and readers know that a reviewer can slant the

student no more than doing homework does.

findings-intentionally or not-in any direction.

Rhetoric, of course, can not solve the debates Box score reviews provide objectivity, but results

over issues such as: "How well does technology must be interpreted in a limited way. These

teach Japanese students?," "Do students feel reviews tell how often one or another approach

depersonalized by methods of teaching that is better, but they do not say how much better,or why one is better. A box score may show

dies,

address: Department of Educational Stu- that an innovative method beats a traditionales, University of Utah, Salt Lake City, Utah.

*4 Present address: Lander College, Greenwood, approach in 35 of 30 studies. But in Glass's

South Carolina.

words, it does not say whether the innovative

13

14

B. J. SHWALB et al.

Table 1. Effectiveness of four types of technology instruction (TI) as reported in U.S. research.Effect size

Other key findingsTechnology

Review

Grade level (No. of studies)SD

(Number of studies)

CAI

C.-L. Kulik,

1-5

0.48

(+) low-ability student (4)J. A. Kulik,

(25)and Bangert (1984)

0.31CAI

Burns and

1-6

0.36

(+) low and high ability students (4)math

Bozeman

(27)(1981) 0.55

CAI

Hartley

K-8

0.42

(+) elementary levelmath

(1977)

(30)

(+) weak design of study0.60

CAI

J. A. Kulik,

6-12

0.32

(+) student attitude (14)Bangert and

(48)

(+) more recent studiesWilliams (1983)

0.42

(-~-) short durationCAI

Burns and

7-12

0.28

(+) disadvantaged studentmath

Bozeman

(12)(1981) 0.55

CAI

Hartley

9-12

0.30

(-) secondary levelmath

(1977)

(3) 0.37

CAI

J. A. Kulik,

13-16

0.25

(+) different teachersC.-L. Kulik and

(54)P. A. Cohen (1980)

0.64 (-) time for instructionCAI

B. J. Shwalb and

Adult

0.44

(-i-) short durationD. W. Shwalb

(4)(1984) 0.49

CAI

C.-L. Kulik,

Adult

0.27

(+) student attitude (6)J. A. Kulik and

(19)B. J. Shwalb (1984) 0.37

TOTAL

K-Adult

0.34

+13 percentile pointsCAI

(222) 0.50

PI

Hartley

K-8

0.19

(+) researcher-made materialsmath

(1977)

(24)

(+) weak design of study 0.75

PI

C.-L. Kulik,

7-12

0.08

(+) retention (16)B. J. Shwalb and

(47)

(-) student attitude (9)J. A. Kulik

0.48

(+) more recent studies(1982)

(-i-) soft disciplinesPI

Hartley

9-12

0.01

(+) more recent studiesmath

(1977)

(16) 0.62

PI

J. A. Kulik,

13-16

0.24

(+) more recent studiesP. A. Cohen and

(56)

no difference student attitude (4)Ebeling (1980)

0.45

no difference student withdrawal (9)PI

B. J. Shwalb and

Adult

0.40

(H-) non-math coursesD.'W. Shwalb

(25)

no difference student attitude (8)(1984)

0.49 no difference student withdrawal (3)TOTAL

K-Adult

0.19

+8 percentile pointsPI

(168)0.52

VBI

P. A. Cohen,

13-16

0.15

(+) more recent studiesEbeling

(65)

(+) doctorate institutionand J. A. Kulik

0.44(1981)

(HI-) different teacherno difference student attitude (16)no difference aptitude correlation (16)

(H) performance skills

Educational Technology in the Japanese Schools

15

VBI

B. J. Shwalb and

Adult

0.49

(+) different teacherD. W. Shwalb

(11)(1984)

0.49

no difference student attitude (8)no difference aptitude correlation (6)

TOTAL

13-Adult

0.20

+8 percentile pointsVBI

(76)0.45

AT

J. A. Kulik,

13-16

0.20

(+) journal reportC.-L. Kulik and

(42)

no difference course completion (22)P. A. Cohen

no difference student ratings (6)(1979)

no difference aptitude correlation (12)TOTAL

13-16

0.20

+8 percentile pointsAT

(42)TOTAL: ALL

K-Adult

0.26

+10 percentile pointsTI USES

(508)Note: CAI=computer-assisted instruction; PI=programmed instruction; VBI=visually-based instruc-

tion; AT=audio-tutorial instruction; K=kindergarten.

method wins "by a nose or in a walkaway." Other meta-analytic reviews of instructionThe scope of a body of literature is lost with followed Hartley's. At the University of Michi-such oversimplified and bland conclusions. gan, researchers have undertaken a systematic

Objective evaluations and rich interpretive review of major technology uses in Americanresults have come from applying the techniques schools. Major findings from this and otherof meta-analysis, a more rigid review of the research is presented in Table 1. It can readilyliterature than either the box score or narrative be seen that with regard to effectiveness, con-method. Meta-analytic review, defined as "an clusions are straightforwards: in Americananalysis of analyses" by Glass (1976), is the schools technology instruction is equally orstatistical integration of findings from a large more effective in conveying educational contentcollection of results from individual studies. than is conventional instruction. The results fromSince its introduction in 1976 the number of the 508 different studies analyzed in Table I givemeta-analytic reviews have grown rapidly, earlier an overall effectiveness index of 0.26. This meansmeta-analytic findings have been verified through that in the four types of technology reported-replication, and the procedure has gained a computer-assisted instruction (CAI), program-widespread reputation as a reliable and powerful med instruction (PI), visually-based instructiontool for quantifying a body of literature. Re- (VBI), and audio-tutorial instruction (AT)-asearchers who conduct meta-analysis first locate typical technology-instructed (TI) studentstudies of an issue by clearly specified procedures. achieved at the 60th percentile on measures ofThen the study features and outcomes are char- learning while a typical conventionally-instructedacterized in quantitative or quasi-quantitative (CI) student achieved at the 50th percentile. Ofterms. Finally, multivariate techniques are used course, in reporting an overall index of effectto describe findings and to relate study charac- for technology instruction some valuable infor-teristics to outcomes. mation is obscured. For example, CAI appears

Meta-analysis was first applied to a review of to be the most effective of the technologies.instructional technology by Hartley in 1977. However, very recent applications of PI andHartley described 89 comparisons from 40 VBI show these technologies to be as effectivearticles reporting the effectiveness of program- in producing learning gains as is CAI. In addi-med and conventional teaching in elementary tion to achievement, outcome measures ofand secondary mathematics classes. She found retention, student ratings and aptitude-achieve-in the typical comparison that programmed ment correlations have been made. But, asinstruction boosted student achievement by 0.11 reported in Table 1, findings in these areas varystandard deviation units. This is an increase in and presently no overall conclusions should beachievement from the 50th to the 54th percentile. made.Hartley also reported that the study publication In Table 1 the results of technology uses are

year correlated 0.39 with size of effect, i.e., recent presented according to technology type. Whenpublications describe programmed instruction the data from Table 1 is rearranged and pre-as more effective than did earlier publications.

sented according to technology effectiveness by

16

B. J. SHWALB et al.

Table 2. Achievement effect sizes for U.S. elemen- has been demonstrated. In the process of syn-tary, secondary, college and adult stu- thesizing this research an unexpected by-productdents.

has been to delimit the definition of effective. InGrade level

Type of

Effect size (ES)

addition to an evaluation of students who weretechnology

(No. of studies)Elementary CAI 0.42 (82)

taught by technology, such questions as How

PI 0.19 (24) long does this method take? Do the studentsTotal

0.37(106)

feel they really understand? are being answered.Secondary CAI 0.31(63) In the words of Azuma (1979) to substantiate

PI 0.06(63) a claim that one method of instruction is betterTotal

0.19(126)

than an alternative method, "empirical evidenceCollege

CAI

0.25 (54)

guarded by technical precautions must be pre-PI

0.24 (56)

sented." The procedures of meta-analysis meetVBI

0.15 (65)AT

0.20 (42)

these requirements and are used in this reportTotal

0.21(217)

to integrate the findings of 128 studies reportingAdult CAI 0.30 (23) technology uses in Japanese classrooms. A

PI 0.40 (25) meta-analysis of Japanese technology uses willVBI 0.49(11) also be a stringent test of the validity of instruc-Total

0.38(59)

tional technology research in the U.S. ValidationNote: CAI=computer-assisted instruction; PI=

would be important to educators, policy-makers,programmed instruction; VBI=visually-based instruction; AT=audio-tutorial in-

and researchers for a number of reasons. Chiefstruction.

among them is that if some degree of effectivenessis validated, instructional technology research

grade level, as it is in Table 2, other aspects of could then be directed away from evaluativetechnology use are more clearly seen. For comparisons and redirected toward the problemexample, data accumulated and analyzed thus of how do we put together optimum instruc-far indicates that technology instruction is most tional systems for meeting different objectives.effective when used with adult learners and

Our report will address these questions: Howelementary school children; raising these groups' effective is technology in the typical comparativeachievement level almost 2/5 of a standard study? Are certain technologies more effectivedeviation unit above their conventionally-in- than others? Does technology have differentialstructed counterparts, i.e. from the 50th per- effects for different students? What are thecentile to the 65th percentile. Secondary school measured effects of technology on achievement,children seem the least benefitted by applications aptitude-achievement correlation, and studentof technology in the classroom. It may be that attitudes? In addition, we will explore whetherif this group of learners' data could be broken the generalizations pertaining to technologydown into junior high (7-9) and high school classroom use in the U.S. are substantiated by(10-12) segments, it would be found that low findings on technology classroom use in Japan.junior high or senior high student achievement The data come from 128 studies reported sinceis dragging down the overall achievement level 1960 and represent the first application of meta-of the secondary school students' data. Breaking analysis to Japanese research findings.the data down in this way seems not to be pos-sible, however, because national definitions of

II. METHODSjunior and senior high school are no longeruniform e.g. middle schools 6-9, junior high The procedures used in this report are thoseschool, 7-8, or 7-9 etc. of the University of Michigan meta-analysis

U.S. technology research has been charac- group headed by Drs. James and Chen-Linterized by a preoccupation with evaluative com- Kulik. Their technique differs from other eval-parisons. The reason for this preoccupations is uators in that a single study, defined as the setapparent: the educational establishment, teach- of results from a single publication, is weighteders, and parents demanded proof of the effective- equally to all other studies. Many other meta-ness of technologies in the classroom. Now, five analysts aggregate multiple effect sizes from onedecades of instructional research has been report. For example, if a study includes threesynthesized and interpreted. At this point it grades and sex as factors, the latter group ofcan be said that effective learning via technology evaluators would aggregate five effect sizes-

Educational Technology in the Japanese Schools

17

one for each grade which included both sexes single most complete report was used. Whenand one for each sex which included all three an experimenter made repeated comparisons ofgrades. For the same study, the University of the same course, data from the most recentMichigan group would sum over grade and sex comparison were used. When more than onefactors to produce one effect size.

achievement outcome was reported, we summedLocating and selecting studies

over the average from each outcome measure toThe first step was to collect a large number of produce one composite score for the study.

studies comparing the effects of technology These procedures insured independence betweeninstruction and conventional instruction. No studies in analysis.comprehensive indices of library holdings exist

Coding study featuresin Japan, neither are there computerized networks The 128 applications of technology variedof holdings between libraries. Therefore, primary along several dimensions. They described dif-sources for studies were found hand-searching ferent types and durations of classroom use.the educational holdings of two of Japan's most Some studies described television instruction inprestigious libraries: the National Institute for science for fourth graders while others reportedEducational Research Attached Library, and the effects of multi-media instruction in thethe Tokyo University Faculty of Education humanities for second-year high school students.Library. At these libraries a total of 160 univer- Some were recently reported in prefecturalsity and prefectural educational research reports/ bulletins while others had been reported 20bulletins were located. Other reference sources years earlier in scholarly journals.were Educational Index (Japanese Language To characterize study features more preciselyEdition), Japanese Periodicals Index, Humanities we defined 15 variables. Five described theIndex, and major reviews of technology use experimental design of the study (Bracht andreports.

Glass, 1968; Campbell and Stanley, 1963) andAn original pool of 1,700 titles was scanned concerned internal and external threats to valid-

for key words, descriptors, and abstracts; 250 ity. Variables related to the course taught,documents warranted closer examination and school level, and the duration of instructionwere photocopied. Each of these documents were described by six variables. Two variableswere read by three people, two of whom were described the type and use of technology, andnative speakers of Japanese. A total of 104 two described publication features. In Table 3documents contained data that could be used features of the 15 study variables are presented.in the meta-analysis. Japanese university and We were unable to categorize every variableprefectural research reports are similar to single- for every study because some studies reportedtopic monographs in the U.S., and some con- unclear or incomplete data. When there wastained more than one suitable report of research. incomplete variable information, the mean valueThese 104 documents then report the data for of a study feature was "plugged" for that report.the 128 different comparisons of TI and CI used The variables described in Table 3 were codedin this study.

independently by two or more coders. Disagree-To be included in the final sample, a study ments between raters were discussed before final

had to meet five basic criteria. It had to: (1) decisions about variable categorizations werereport a comparative investigation between a made. The lowest inter-rater reliability coeffi-traditional lecture/discussion control group and cient was 0.83 and the median coefficient wasa technology taught experimental group; (2) 0.92.be conducted in j an instructional setting with

Quantifying study outcomesnon-handicapped students; (3) measure achieve- The next step was to express outcomes of eachment and/or performance outcome in quantifi- study in quantitative terms. First, we describedable units for both the TI and CI groups; (4) the effect of TI on achievement in such a waybe methodologically sound; and (5) be available that a positive result means the findings favorin its entirety for inclusion (no data are included the treated (TI) group, while a negative findingin analysis based on abstracted information).

occurs only when the control (CI) group im-Other guidelines insured that a single research proves more than the treated groups. Three basic

effort would not be counted more than once in indices of achievement outcome effect werethe overall analysis. When several articles or calculated: (1) the average examination scoreexperimenters reported the same comparison, the expressed as a percentage in a TI class minus

18

B. J. SHWALB et al.

Table 3. Study features of technology-use reports with achievement data.

Effec t sizes

Percent differences

Coding category

Number of

ES(SD)

Number of

/Dif (SD)comparisons

comparisons

Equivalence of Comparison GroupsNo evidence of equivalence 13 0.48(0.34) 13 12.4(15. 1)Evidence of equivalence 100 0.39(0.30) 104 7.6(5.7)Random assignment

3

0.61(0.29)

3

11.4 (3.4)

Control for Instructor EffectDifferent instructor

61

0.45 (0.29)

63

9.1(8.1)Same instructor

55

0.36(0.32)

57

7.3 (6.5)

Control for Historical EffectInstructed at different times

8

0.30(0.41)

10

6.0(8.9)Instructed at same time

108

0.42 (0.30) 110 8.4 (7.3)

Scoring Bias in Criterion TestNon-objective

16

0.47 (0.27)

18

9.7 (4.5)Objective

100

0.40(0.31)

102

8.0(7.8)

Research Bias in Criterion TestResearcher developed test 54 0.44 (0.31) 56 9.5 (9.0)Collaborative test 53 0.40 (0.31) 55 7.2 (5.4)Commercially standardized test

9

0.27 (0.29)

9

6.7 (6.2)

Materials TaughtIntroductory

107

0.41(0.31)

111

8.3 (7.6)Others

9

0.42(0.28)

9

7.5(3.9)

Course Content AreaSocial science 21 0.41(0.39) 23 10.5 (12.7)

Language 26 0.38 (0.25) 26 7.1(5.0)Science 26 0.40 (0.32) 26 7.1(5.4)Mathematics

43

0.43 (0.30)

45

8.5 (6.0)

Ability Range of StudentsRestricted

14

0.45 (0.28)

14

9.3 (6.5)Unrestricted

102

0.40 (0.31)

106

8.1(7.5)

Duration of InstructionLess than a semester 96 0.40(0.31) 99 6.5(6.9)One semester 9 0.47 (0.29) 9 7.7 (5.6)More than a semester

11

0.40(0.34)

12

5.8(7.5)

Hours of InstructionLess than 1 54 0.41(0.31) 57 8.1(5.5)Between 1 and 6 20 0.42 (0.34) 20 7.7 (6.9)More than 6

42

0.41(0.30)

43

8.7 (9.6)

Grade LevelElementary school 36 0.42(0.40) 37 8.6(3.4)Junior high school 69 0.40(0.25) 70 8.8(9.0)High school 11 0.48 (0.39) 12 6.5 (7.6)College

0

0

(0)

1

12.9 (0.0)

Type of TechnologyProgrammed instruction 44 0.50 (0.24) 47 8.9 (3.8)Audio-tutorial instruction 6 0.09 (0.27) 6 7.5 (5.9)Visually-based instruction 36 0.39(0.38) 37 10.2(9.7)Computer-assisted instruction 4 0.45 (0.38) 4 13.3 (0.7)Multi-med ia instruct ion

26

0.35 (0.34)

26

5.3 (8.5)

Technology UseSupplement

45

0.39 (0.36)

45

8.0 (10.0)Substitute

71

0.42(0.27)

75

8.4(5.2)

Source of StudyEducational research journal 9 0.57(0.40) 10 15.2(17.4)Prefectural/municipal research facility 68 0.42 (0.30) 68 8.3 (4.9)Individual school 12 0.40(0.33) 12 9.6(7.8)University research center 21 0.24 (0.25) 23 3.9 (5.8)Public educational research institute

6

0.59(0.27)

7

9.1(4.3)

Educational Technology in the Japanese Schools

19

Year of Study1960-1967 31 0.38 (0.27) 31 7.8 (5.4)1968-1973 47 0.45 (0.30) 49 9.5 (8.9)1974-1982

38

0.38 (0.45)

40

7.0 (6.6)

Table 4. Intercorrelations of four effect measures

allows effect sizes based on different dependentof achievement.

variables and different studies to be compared.Unit and course student achievement (N=116)

Regression equations were used to fill in% Cohen's Glass'sscores

d

ES

missing data on specific effect sizes. The regres-Difference in scores sion equations were based on the correlations of

Cohen's d 0.95 four measures of effect sizes-percent difference,Glass's ES 0.95 0.99 ES, d, and a 4-category scale based on directionFour-category scale

0.83

0.86

0.87

and significance of difference. These variousHigh-, middle-, and low-aptitude student

measures of effect size correlate very stronglyachievement

% Cohen's Glass's

when applied to the same data set making itscores

d

ES

possible to write regression equations with aHigh (N=27): % scores

high degree of accuracy from one kind of mea-•

0.91 sure of effect to another. Table 4 presents the•

0.90 0.99 intercorrelation matrices for the four measures4-category 0.78

0.79

0.79

used in the present analysis to fill in missingMid (N=27): % scoresd

0.97

observations in studies with an incomplete report•

0.97 0.99 of results.4-category 0.85

0.84

0.85

We also examined the effect of TI on theLow (N=27): % scores 0.97

correlation between student aptitude and achieve-•

0.97 ment. The significance test of the differenceES

0.97

0.994-category 0.72

0.76

0.78

between is is accomplished via the Fisher zSex differences in achievement

transformation, i.e. z=arctan r, and the effect% Cohen's Glass's

size is q=zl-z2 (Cohen, 1977). Owen (1965)scores

d

ES

was the source of both the z transformation andMale (N= 12): % scores

the normal curve values.d

0.87

Finally, we quantified TI effects on attitudeES

0.87

0.99

toward method of instruction and attitude to-Female

0.69

0.82

0.84Female (N= 12): % scores

ward subject matter. Two lists of model rating•

0.95 items for the two attitude measures were devel-•

0.97 1.00 oped. If a study had data for any item that4-category 0.79

0.77

0.79

appeared in either list, the data were used inRetention achievement (N=34)% scores

analysis. All attitude ratings were converted to•

0.92 a 7-point scale, where 7 represented the highest•

0.89 0.98 rating (i.e. high quality, enjoyment, learning,4-category 0.66

0.75

0.72

etc.), and 1 represented the lowest possible rating.We also calculated ES and d for these instruc-

the average in the comparable conventional tional outcomes.class; (2) Cohen's (1977) statistic, d, a devia-tion-unit measure of oeffect size, defined as the

III. RESULTSdifference between the means of the two groupsbeing compared divided by the standard devia-

Findings are described for three areas: (a)tion common to the two populations; and (3) student achievement; (b) aptitude-achievementGlass's (1976) statistic, ES, a similar pure

correlation; and (c) student attitudes.measure of effect size, defined as the differencebetween experimental and control groups di- 1. Student Achievementvided by the standard deviation of the control

Overall unit and course student achievementgroup's raw scores (as opposed to some stan- Examination performances were reported indardized error). This means that effect sizes are 116 studies. Student performance was superiorin a z-score unit. Eliminating the scaling factor in TI classes in 99 of these studies. In the remain-

20

B. J. SHWALB et al.

ing 17 studies, CI students' achievement was statistically significant, t(115)=5.42, p<0.01,superior. Fifty-one studies reported that differ- and favors the effectiveness of technology overences between teaching methods were statistically conventional instruction.significant. Of these, 48 favored TI and 3 favored

Technology effectiveness was also interpretedCI. If there were no systematic effect of method in reference to the standard normal curve byof instruction on student achievement, about examining the mean of Glass's ES. In 116half the results would favor one method and studies, mean ES was 0.41. This is the same ashalf would favor the other method. Instead a saying that in the typical TI class, a student'sclear majority of studies favored TI. The null achievement was raised about two-fifths of ahypothesis of no difference between instructional standard deviation unit. Another interpretation

methods can be rejected, gz(l, N=116)=23.8, of the ES statistic is that a typical TI studentp<0.001.

performed at the 66th percentile on examinationsThe effect of technology was measured more while the typical CI student performed at the

precisely than tallying. Achievement percentage 50th percentile. Effects of this magnitude aredifferences on examinations given to technology described as too small to be verified withoutand conventionally taught students were also special measurement procedures (J. Cohen, 1977).examined. The effects of TI in the typical study For example, a difference in heights of 2 cmwere small, but there were some notable varia- (approximately the same as an ES of 0.41)tions from study to study. In a study by Matsuo normally needs to be verified by measurement.(1971), fourth graders used VBI to study social An effect is said to be medium when ES=0.5sciences. In this study the TI group outperformed and large when ES= 0.8.their CI peers by almost 60 percentage points

Next we more closely examined study featureon a test of social science achievement. However, variables. First ANOVAs were conducted toa study by Yasuhiko and Sakurai (1977) reported determine if there were any significant withinthat conventionally instructed eighth graders variable differences. One of the variables usedbettered their technology instructed peers by to describe publication features, source of study,almost 11 percentage points when both groups had an F ratio of 2.82, p<0.03, df=1,116. Thiswere tested at the end of a social science unit means that differences among categories withinthat had been taught to the experimental class this variable cannot be attributed to random

via VBI. In Fig. 1 a distribution of percentage variation. Individual schools and prefectural/

differences between TI and CI classes is presented. municipal research facilities publish reports in

This figure shows there is a difference plateau which the effectiveness of technology is assessedof between five and 10 percentage points. The by ES to be about 0.41, i.e. the estimated ESoverall mean difference between the TI and CI mean value. Studies published at universitygroups was 7.31%, sd=6.0. This difference is research centers report technology uses in the

so classroom that are considerably less effective(ES=0.24) than the average while studies pub-

s •

lished in either educational research journals or

4 0 .

at public educational research institutes report

^35 . technology uses in the classroom considerablymore effective (ES=0.58) than average. No

=30

other F ratios with p<0.1 were found.r25

Next we examined whether study featurescorrelated with achievement ES. Two study

zo _features, control for historical effects and mate-

15 . rials taught, could not explain variation in

10 .

student achievement outcomes because therewas little variation attributable to these features.

s '

For this reason these two features were left out0 of further analysis. The 13 remaining indepen-to -E

0

5DIFFERENCE;

20TI

25

OVER0

CI35 4o

PERCENT DIFF

dent variables were then included in regression

Fig. 1. Percentage difference in achievement for analysis. No variable correlated with overall

116 studies. TI=technology instruction; CI=con- achievement effect size at p<0.1. To investigateventional instruction.

the possibility that a combination of variables

20

18

16

W14

X12HU- 100

Wm

2

0-8 -4

0

4

8 12 16PERCENT DIFFERENCE; TI OVER

TECHNOLOGY TYPE

PROGRAMMED

VISUALLY-BASED

MULTI-MEDIR

Fig. 2. Achievement differences for three types oftechnology instruction (TI). CI=conventional in-struction.

20 24CI

28

Table 5. Achievement effect sizes for elementary(Elem), junior high (JH), and high school

(HS) students.Technology School

ES

SD

Ntype

level

PI

Elem

0.44

0.15

15JH

0.41

0.20

24HS

0.92

0.00

1VBI

Elem

0.49

0.25

11JH 0.49 0.36 18HS

0.52

0.30

4MM

Elem

0.46

0.35

7JH

0.24

0.44

13HS

-0.02

0.26

3CAI

Elem

-

-

0JH

0.70

0.06

3HS

0.84

0.00

1AT

Elem

-

-

0JH 0.37 0.29 4HS

0.33

0.04

2TOTAL

Elem

0.46

0.23

33JH

0.41

0.30

62HS

0.40

0.19

11Note: No CAI or AT elementary school studies

were located. P1=programmed instruc-tion; VBI=visually-based instruction;MM=multi-media instruction; CAI=computer-assisted instruction; AT= audio-tutorial instruction.

might predict effect sizes more accurately thansingle predictors, a stepwise multiple regressionanalysis was carried out. No combination ofvariables produced a model significantly relatedto effect size.

Unit and course student achievement withintechnologies

Our next analysis concentrated on studentachievement within each of the five technologiesused in the gathered research reports. In Table5, effect sizes by school levels are given for eachtechnology use.

Programmed instruction. PI was used as acomparison method in 44 studies. It was reportedin 41 studies that students using PI had betterexamination performance than students taughtconventionally. Two studies reported CI studentsoutperformed PI students. In one study bothgroups had the same score. The proportion ofstudies favoring TI over CI is significant, x 2 (1,N=44)= 16.96, p<0.01. A null hypothesis of noeffect for type of instruction is rejected. Signi-

ficant differences were reported in 22 of the 44studies and all favored TI.

PI achievement was also examined usingcontinuous achievement scales. Examinationachievement differences between TI and CIstudents was 8.73 % points, sd=7.08. A matched

pairs t-test on the difference of the means is

not significant, t(43)=1.22, p<0.3. Amongstudies of PI there were notable achievementvariations. The biggest achievement differencewas reported in a 1973 study by Noguchi. Inthis study eighth grade social studies PI studentsoutscored CI students by 31 % points. The leasteffective use of PI was reported by Baba andYoshimura (1982). In their study, third graderswho were taught math using CI outscored PIcounterparts by almost 7 percentage points.Figure 2 presents a distribution of percentagedifferences between PI and CI classes.

PI studies reported the most effective use oftechnology. Glass's ES for the 44 studies was0.50, sd=0.46. In the typical PI class then, astudent's achievement was raised one-half of astandard deviation unit. This means a typical

PI student scored at the 69th percentile onachievement measures while a typical CI studentscored at the 50th percentile. J. Cohen (1977)describes effects of this magnitude as medium,i.e. performances would be discernable by ateacher even before achievement measurementswere made.

Correlation of study features to PI achieve-ment outcomes showed the study feature equiv-alence of comparison groups was modestly cor-related to achievement, r=0.29, p<0.07. An

Educational Technology in the Japanese Schools

21

22

B. J. SHWALB et al.

ANOVA revealed that studies in which groups among VBI studies, r= -0.34, p<0.05. When

were randomly assigned to treatments reported the same instructor taught both the VBI and

the largest effect sizes, while studies in which the CI class, learning gains were not as pro-

there was no equivalence between groups re- nounced for the VBI students as they were

ported the smallest effect sizes. Stepwise regres- when different instructors were used to teach

sion analysis produced no model of variables the VBI and CI class. This correlation reflects

that would better predict achievement than did the same phenomena as does the first reported

the single study feature.

correlation, i.e. less stringent experimental design

Because 15 PI studies were conducted with resulted in greater achievement for VBI students

only elementary school pupils, and 24 PI studies than did more stringent experimental design. No

reported effectiveness at the junior high school combination of variables better predicted

levels, we conducted technology by school level achievement than did equivalence of comparison

multiple correlation analysis. However, no study groups and control for instructor effect.

variables correlated at p<O.l with PI achieve-

Eighteen VBI studies were conducted in junior

ment ES for elementary or junior high students. high school classrooms. This N is sufficient for

Additionally, no combination of variables pre- correlational analysis. Analysis showed three

dicted achievement at either the elementary or variables correlated strongly with ES: equiv-

junior high level.

alence of comparison groups, r= -0.58, p<0.01;

Visually-based instruction. VBI was the second control for instructor effect, r= -0.50, p<0.03;

most commonly used technology type among and source of study, r= -0.46, p<0.05. The

the reports we found. Achievement differences first two variables reflect the reported finding

favored the technology group in 31 of 35 studies; for all VBI studies, i.e. less stringently controlled

in the remaining 4 studies achievement differ- comparisons were more effective than highly

ences favored CI. In these 35 studies, 14 reported controlled comparisons. The third variable,

differences that were significant; 13 favored VBI source of study, suggests that highly successful

and one favored CI.

uses of VBI with junior high school learners

The mean achievement difference between the have been reported by public educational re-

35 VBI and CI classes was 7.71 % points, sd= search institutes and university research centers.

11.71. This mean is not significantly different Unsuccessful uses have been reported in educa-

from zero, t(34)=0.65, p<0.6. The largest tional research journals. These three single

variations in achievement were found within variables predict achievement better than a

studies using VBI, which accounted for both combination of variables.

the top and bottom of the achievement range Multi-media instruction. MM was the third

among all studies. Variations in achievement most commonly reported technology use. Usu-

between VBI and CI students are presented in ally it involved combining audio and visually-

Fig. 2.

based instruction and at times programmedVBI effectiveness was also interpreted in refer- materials were also used with these other meth-

ence to the standard normal curve. ES for 36 ods. Achievement differences were reported instudies was 0.39, a small effect. This is interpreted 25 studies. MM students performed better than

to mean that typically VBI students outper- CI students in 18 studies, and CI students per-

formed CI students by 2/5 of a standard devia- formed better than MM students in 7 studies.

tion unit. This means that while an average CI The distribution of differences is not significant,

student achieved at the 50th percentile, an aver- ga (1, N=25)=2, p<0.2. Significant differences

age VBI student achieved at the 65th percentile.

were reported in 13 studies; 12 favored MM

Two study features significantly correlated to and one favored CI.

VBI achievement. Equivalence of comparison

In 25 studies the mean achievement difference

groups was negatively related to achievement, r= was 5.32% points, sd=7.0, t(24)=0.75, p<0.5,

-0.35, p<0.04. ANOVA showed that among a nonsignificant finding. A hypothesis of no

VBI studies, no evidence of equivalency was effect of instruction is accepted. Again wide

associated with a large ES mean of 0.83, i.e. variations in treatment effects were found. In a

less stringently constructed experimental designs fourth grade social science class (Kitakyushu

resulted in greater gain than did more exacting Municipal Educational Research Institute, 1975),

experimental designs. Control for instructor effect MM instruction was significantly less effectivewas also negatively related to achievement ES than CI (by 10 percentage points). Fuchu and

Educational Technology in the Japanese Schools

23

Kotaki (1974) report an investigation with tenth students in Aichi Prefecture. ES for six studiesgraders studying mathematics. In this study, the was 0.09. This means a typical AT student'saverage MM student gained 24.4 percentage performance was less than one-tenth of a stan-points over an average CI student. A distribu- dard deviation unit above that of a typical CItion of the percentage differences between MM student. A typical AT student achieved at theand CI classes is presented in Fig. 2. 54th percentile, while a typical CI student

MM instructed students achieved at the 64th achieved at the 50th percentile. Too few studiespercentile on tests of achievement and their CI were available for either correlation or regres-counterparts achieved at the 50th percentile.

sion analysis.This is a performance difference of almost 2/5 Computer-assisted instruction. Only four com-of a standard deviation unit. The mean ES for parative CAI studies were located. Three favored26 MM studies was 0.35, a small difference.

CAI and one favored CI. Two studies reportedStudy features were correlated to achievement statistically significant differences and both

for MM studies. Three variables related signif- favored CAI. Mean achievement difference wasicantly to ES. They were (1) scoring bias in 4.82 percentage points, sd=5.36. This is notcriterion test, r= -0.37, p<0.04; research bias significant, t(3)=0.78, p<0.9. The range ofin criterion test, r= -0.37, p<0.08; and grade achievement was from 11 to -1 percentagelevel, r= -0.37, p<0.08. ANOVAs revealed the points. The greatest achievement differencefollowing: (1) non-objective tests resulted in (Ariyoshi, 1972) used CAI to teach mathematicssignificantly higher achievement for students to eighth graders. The least effective use of CAIinstructed with MM, ES=0.63, than did objec- was reported by the Kagawa University CAI Re-tive tests, ES=0.17; (2) MM students scored search Group (1970). This study also used CAIbetter on researcher-made tests, ES=0.43, than to teach mathematics to eighth graders. ES forthey did on collaboratively made tests, ES=

the four CAI reports was 0.45. This is a separa-0.16, or on commercially available tests, ES=

tion of almost one-half standard deviation. It-0.16; and (3) the most effective use of MM for means a typical CAI student performed at thestudents was in elementary schools, ES=0.46. 67th percentile and a typical CI student per-At junior high schools, MM was still somewhat formed at the 50th percentile. Too few studiesmore effective for students than CI, ES=0.24, were available for further analysis.and at the high school level MM instructionhad a negative effect on students' performance, 2. Unit and Course Student Achievement withinES= -0.2. Once more, single variables pre- School Levelsdicted better than did a model combination of Perhaps learning via technologies is not sovariables in stepwise regression. Too few MM much related to the particular technology as itstudies were available for media by school level is to particular learner characteristics. Mostcorrelational analysis. technologies, for instance, have in common

Audio-tutorial instruction. We found only six basic pedagogical features, i.e. behavioral ob-studies that reported achievement for both AT jectives, active student response, and positiveand CI students. Three studies favored AT reinforcements. It may be that learner charac-instruction and three favored CI. One would teristics, such as age, act on these commonexpect half the studies to favor one method and features in different ways. Therefore, we analyzedhalf to favor the other method if there were no student achievement at different grade levels.difference in method. Two studies reported Elementary schools. Thirty-five studies reportfinding significant difference, one favored AT the achievement of elementary school childrenand one favored CI. who used technology instruction-28 favored TI

The mean achievement difference between and 7 favored CI, X 2 (1, N=35)=5.71, p<0.02.AT and CI classes was 2.07 percentage points, Ten studies reporting significant achievementsd= 6.54. This mean is not significantly different differences favored the TI groups; none favoredfrom zero, t(5)=0.29, p<0.9. The range of the CI groups. Students in elementary schoolachievement difference was from 14.34 to -7.1 classes typically gained 7.24 percentage points,percentage points. Both the high and the low sd=11.2, more from TI than did CI students.in the range of AT scores were reported from This difference is not significant, t(34)=0.64,the same group of researchers (Wakai et al., p<0.8.1977), using language courses for high school

The mean ES found for 36 elementary school

studies was 0.42, sd= 0.40. This indicates that atypical elementary school child using TI scoredtwo-fifths of a standard deviation unit higherthan an average elementary school child whohad used CI. Another interpretation is thattypically a TI child scored at the 66th percentilewhile an average CI child scored at the 50thpercentile.

Year of study had a modest correlation toelementary school ES, r=0.37, p<0.04. AnANOVA indicated ES=0.36 for the years 1960-1967; ES=0.48 for the years 1968-1973; andES=0.56 for the years 1974-1982. Obviouslymore recent studies report higher achievementgains for TI students than do less recent studies.

Junior high schools. Sixty-eight studies reportedachievement scores for both TI and CI studentsat the junior high level. Fifty-nine studies favoredTI achievement and nine favored CI achieve-ment. Significantly more studies favored theuse of technology, x 2 (1, N=68)=17.65, p<0.01. Thirty-two studies reported significantachievement differences. Of these 31 favored TIand one favored CI. Junior high school studentswho received their instruction via technologytypically scored 6.99 percentage points, sd=7.56, more on their examinations than did juniorhigh CI students. This does not represent asignificant gain by the TI group, t(67)=0.92,p<0.4.

ES was calculated for 69 studies, mean ES=0.4, sd=0.25. We interpret this to mean thatTI raised the achievement of junior high schoolstudents above that of their CI counterparts bytwo-fifths of a standard deviation unit. That is,while CI students scored at the 50th percentilein testings, TI students scored at the 66th per-centile. No single study feature and no combina-tion of features predicted junior high schoolachievement.

High schools. Only 11 studies had achievementdata from high school reports-10 of thesefavored TI and one favored CI. The differencewas significant, x 4 (1, N=11)=2.91, p<0.1.Seven studies favored TI significantly and onesignificantly favored CI. Mean achievementdifference favored TI by 8.18 percentage points,sd=7.94, which is not significant, t(10)=0.98,p<0.4.

The achievement effect size was also reportedfor 11 studies; ES=0.48, sd=0.39. This signifiesthat high school students using TI typically out-scored their CI peers by one-half of a standarddeviation unit. A TI student would average a

percentile ranking of 68 while a CI studentwould average a percentile ranking of 50. Therewere too few studies for further analysis.

Colleges. One study reported a difference of12.92 percentage points between TI and CIstudents. No college studies reported an ES.

High-, middle-, and low-aptitude student achieve-ment

Twenty-seven studies reported separate datafor high-, middle-, and low-aptitude students.These achievement measures were dependentwithin studies, i.e. each study provided high-,middle-, and low-aptitude achievement percent-ages and ES, so data were analyzed using re-peated-measure procedures.

High-aptitude students. Examination perfor-mance was higher in TI classes in 17 studies andfor CI classes in 10 studies. Two studies reportedfinding significant differences; one favored TIand one favored CI. We accept a null hypothesisof no difference between teaching methods, x2

(1, N=27)=0.67, p<0.5.The effect of technology was also measured

using percentage point differences. The averagedifference between the TI and CI students was2.1 percentage points, sd=5.8, a non-significantfinding. Again the effects of TI in the typicalstudy were small and differences varied fromstudy to study. Variations in achievement rangedfrom a most effective use by the Kitakyushu

20

1s

1s

W 1'1

=I2rX100

W

Z

8

6

4

;-20 -15 -10 -5

0

5 10 15 20 25PERCENT DIFFERENCE; TI OVER CI

APTITUDE LEVEL

30

LOW

MIDDLE

HIGH

Fig. 3. Achievement differences for three levels ofaptitude. TI=technology instruction; Cl=conven-tional instruction.

Educational Technology in the Japanese Schools

25

Municipal Educational Research Institute (1975) ment ES for the 27 studies was 0.35. Anotherfor 10 hr of MM instruction with fourth interpretation of ES from the normal prob-graders (+56 percentage points) to -15 per- ability curve is that the overall middle-aptitudecentage points, the least effective use with high- control student achieved at the 50th percentileaptitude eighth graders for 25 hr of math PI and the average middle-aptitude TI student(Kagawa Prefectural Shigumo Junior High achieved at the 64th percentile.School, 1971). Achievement differences for high-

The ES difference between middle-aptitudeaptitude TI and CI students are presented in TI and CI students (0.35) and the overall ESFig. 3.

from these 27 studies (0.31) is small and non-The average ES for high-aptitude students significant, ES=0.04, t(26)=0.08, p<0.9. None

was 0.12, while the average ES for all students of the 13 independent variables included in theenrolled in classes from which high-aptitude regression analysis correlated significantly withstudent data was taken was 0.31. The standard the dependent variable of middle-aptitudedeviation for these two effect sizes was taken achievement effect size. No combination offrom the unrestricted total variance of all stu- variables correlated to predict middle-aptitudedents in the 27 studies. The effect size difference, achievement effect size in a multiple regressionES=-0.19, is significant, t(26)=2.13, p<0.05.

analysis.It reflects that there is substantial difference in

Low-aptitude students. In 16 of 27 studies,achievement gain between groups.

low-aptitude students scored higher on achieve-Perhaps there were systematic difference be- ment tests than CI students. In the remaining 11

tween high-aptitude studies reporting large and studies, low-aptitude CI students scored higher.small effects that were due to the relationship Statistically significant differences were reportedbetween high-aptitude achievement and study in seven studies; five favored TI and two favoredfeatures. Regression analysis showed no study CI. The null hypothesis of no difference infeature was significantly related to high-aptitude achievement performance between low-aptitudeachievement ES. A stepwise multiple regression TI and CI students is accepted.analysis showed no combination of study fea-

The mean achievement difference betweentures could be used as a predictor model.

low-aptitude TI and CI students was 3.7 per-Middle-aptitude students. Achievement of centage points, sd= 8.5. This difference is non-

middle-aptitude students was superior for TI significant. Achievement variations are presentedclasses in 21 of 27 studies. The remaining six in Fig. 3. The range of achievement was from astudies showed superior achievement in CI gain of 17.5 percentage points (Hirayama andclasses. Significant achievement differences were Satoh, 1975) to a loss of 8 percentage pointsreported in seven studies; six favored TI and (Konno et al., 1973) for low-ability technologyone favored CI. The null hypothesis of no users.effect of technology instruction on middle- The mean ES was 0.25. On the standardaptitude student achievement is rejected, x2 normal curve this would be interpreted to mean(1, N= 27)= 14.87, p<0.01.

that low-aptitude TI students scored at the 60thAgain we looked with more precision than percentile while their comparison group of CI

just tallying. The average examination score students scored at the 50th percentile. Thedifference for middle-aptitude students was 6.3 difference between low-aptitude ES and overallpercentage points, sd=8.1, which is nonsignif- ES is not significant, t(26)=-0.81, p<0.5.icant, t(26)=0.98. Variations were wide among

Next we examined the differences in achieve-middle-aptitude learners. Most effective (+24 ment ES of paired comparisons between low-percentage points) was a 1971 study reported by (0.25), middle- (0.35), and high- (0.12) aptitudethe Kyoto Municipal Educational Research students to a hypothesized mean difference ofInstitute. In that study eighth graders received zero. There is a significant difference between5 hr of MM instruction in math. Least effective middle- and high-aptitude student ES, t(26)=(-9 percentage points) was 3 hr of MM English 2.2, p<0.05. This means that technology isinstruction given to eighth graders (Chiba most effective in raising the achievement ofPrefectural Educational Research Center, 1974). middle-aptitude learners and significantly lessAll variation differences are presented in Table 3. effective for high-aptitude learners.

ES was calculated using the unrestricted

Study feature variables were regressed on thestandard deviation. Middle-aptitude achieve- dependent variable low-aptitude achievement

26

B. J. SHWALB et al.

effect size. Two variables modestly correlated to p<0.01. Both for achievement and retentionES: (1) equivalency of comparison groups, r=

achievement, 13 studies reported that the dif--0.30, p<0.09, and (2) scoring bias, r= -0.32, ferences between technology and conventionalp<0.08. As earlier it appears that less rigorous instruction were statistically significant. For bothexperimental design results in higher reported types of achievement, 11 favored TI and 2achievement than does more rigorous experi- favored CI. These differences are marginallymental design. No combination of the 13 in- significant, x 2 (1, N=34)=3.11,p<0.1.dependent variables predicted ES better than

The average TI student scored 3.1 percentagedid these two study features.

points more than the average CI student onSex differences

retention examinations, sd=4.9. The ES forAchievement data were dependent within retention achievement was 0.22; for the same 34

studies, i.e. each study provided both male and studies, achievement ES was 0.31. One wouldfemale effect sizes, so data were analyzed using expect that with the passage of time studentsrepeated measures procedures.

forget some information, and this predictableMale students. Twelve studies reported sepa- change is reflected in the decline of the ES

rate data for male TI and CI students; ten showed value. More importantly, on tests of retention,superior TI performance and two showed as on tests of achievement, TI students outscoresuperior CI performance. The two studies CI peers. One month after learning, the meanreporting significant differences favored the TI interval before re-testing in the 34 retentiongroup. The percentage point difference between studies, the average CI students tested at thegroups was 3.8, sd=4.7, a nonsignificant finding. 50th percentile while the average TI studentThe unrestricted variance was used to calculate tested at the 58th percentile. Study features dideffect sizes. The effect size for male achievement not significantly correlate to retention achieve-was 0.23. We interpret an ES of 0.23 to mean ment ES in either multiple or stepwise regression.that CI males performed at the 50th percentilewhile TI males achieved at the 59th percentile. 3. Aptitude-Achievement CorrelationFor the same 12 studies, the ES for all students

In nine studies investigators examined thewas 0.27.

effects on achievement of both teaching methodFemale students. Female TI students achieved and student aptitude. Correlations of student

significantly better than did their CI counterparts aptitude to student performance were measuredin two of the eight studies which favored TI in somewhat different ways in these studies. Sixover CI. Three studies favored the achievement studies correlated students' performance on theof CI females, and one study reported no differ- same test given before and after instruction.ence between female groups. The achievement Two studies correlated IQ scores to studentdifference in percentage points between TI and achievement. One study correlated students'CI females was 5.1, sd=8.2, a nonsignificant previous grades in the subject taught to theirfinding. ES was 0.23, meaning a typical female final performance. The mean aptitude correla-TI student achieved at the 59th percentile while tion for TI students was 0.41 and for CI studentsa female CI student achieved at the 50th per- it was 0.53. The expected correlation for bothcentile. For both male and female achievement the TI and CI group is 0.50. Neither groupdata, there were insufficient studies for regression differed significantly from the expected value.analysis.

A value which designates the degree of differ-Retention achievement

ence for aptitude correlations between groupsHow well do students remember what is is Cohen's q. The q value difference between TI

taught to them? Tests of retention, given some and CI groups is -0.15, a small and nonsig-time after the instructional period, should nificant finding.provide the answer to this question. Thirty-fourstudies provided retention achievement data- 4. Student Attitudes25 favored TI and 9 favored CI. This difference Rating toward subject. Only eight studies con-is marginally significant, x2 (1, N=34)=3.76, tained student ratings of the subject taught forp<0.1. For the same 34 studies, 28 had favored both TI and CI students. TI ratings were higherTI and 6 had favored CI on measures of achieve- in 6 of the 8 studies and CI ratings were higherment made immediately after learning. This in the other two studies. Two studies showeddifference is significant, x2 (1, N=34)=7.12,

statistically significant differences in favor of TI

Educational Technology in the Japanese Schools

27

and one showed a significant difference favoring the achievement gains of TI students are notCI. The average difference in quality rating was spectacular. The gains would probably not be0.61 on a 7-point scale from 7 (highest rating) noted without the help of special measuringto 1 (lowest rating). In the typical course, the tools. These gains, however, are substantiallyTI rating of course quality was 4.39, and the greater than those reported in the typical studyCI rating was 3.78. This difference is small and of American technology instruction. Becausenonsignificant. It corresponds to an ES=0.12.

conventional teaching is so standardized andRating toward method. Ratings by students of uniform in Japanese classrooms, the effect of

their attitudes toward instructional method were technology has probably been measured withavailable in 11 studies. TI ratings were higher less noise than is possible in a typical Americanthan conventional ratings in 6 studies, CI rat- classroom where conventional teaching mayings were higher in 4 studies, and in one study not be the norm of classroom activity.ratings were identical. Only one study showed a

Technology instruction did not have a signif-statistically significant difference, and it favored icant effect on other outcome variables studiedTI. Again a continuous measure of effect size in this meta-analysis. Students' reactions towas applied to the data. The average difference classes taught with TI were not discerniblyin quality ratings was 0.17 on a 7-point scale. different from reactions of CI students. StudentsThe TI mean rating of course quality was 4.27, gave the two kinds of classes similar ratings ofand the CI mean rating was 4.10. This corre- overall quality, and reported about the samesponds to an ES of 0.10, which is nonsignificant amount of learning, enjoyment, and work in theand small.

two kinds of classes. Attribute-achievement cor-relations were also similar with and without TI.

IV. DISCUSSION

In both types of classes, the expected correlationof about 0.50 was found. In 34 studies of reten-

The use of meta-analytic techniques makes it tion, 62% reported no significant differencespossible to reach more exact conclusions about between TI and CI, and 84 % of the studiesthe effects of technology instruction than do reporting significant differences favored TI.other reviews of research. The greater precision

In the results of our analysis, we also foundin conclusions about the effects of technology a general tendency for programmed learning toinstruction is evident in each of the four major be the most effective technology used. There wasareas that were investigated: (1) overall effects also a trend for middle-aptitude students toof technology instruction, (2) the relationship benefit the most from technology instruction.between study features and study outcomes, (3) There were indications that students who scoredresults of attribute-treatment and multi-factor well on examinations after being taught by TIstudies, and (4) student attitudes toward tech- remembered the course materials as well as CInology instruction.

students. Our data also supports the idea thatFirst, we considered the overall effects of technology is as effective with boys as with

technology instruction. In a dox-score review of girls.the 116 achievement studies, we found that 56 % We can be more confident of our conclusionsof the studies of student achievement reported about TI's effects on achievement than we areno significant differences between technology about its effects in other areas. Educationaland conventional teaching, and 94% of all researchers study the effects of TI on achieve-studies reporting significant difference favored ment often, and they reported achievementtechnology lgarning. results fairly completely. In comparison, they

Traditional reviews, however, seldom mention neglected to study the effects of TI in otherthe size of experimental effects. Meta-analysis areas. Only 19 comparative studies examinedadds to our knowledge of experimental effects student reactions to instruction, for example;by focusing on their size as well as significance. only 12 studies reported student achievement byIn the typical study, the size of TI effects was sex; and only nine studies reported on attribute-small. The average ES in the achievement treatment interactions. And the data providedstudies in this meta-analysis was 0.41. This in these studies were given in the most sketchycorresponded to an examination gain of about form. Few studies gave any data on the amountseven and one-third percentage points, or a of variation within groups for study time, onpercentile gain of 16 points. By most standards, cost effectiveness of different technologies, or

28

B. J. SHWALB et al.

on teacher preparation time to implement that was given to different studies in the analysis.

instruction. This meta-analysis gave equal weight to each

Another set of analyses reported on the independent study that was located, whererelationship of study features and study out- "study" was defined as a set of results described

comes. In overall analysis, one relationship in a single report or publication. If studies had

between design features and study outcome was been weighted according to the number offound. Findings in published studies and from students, classes, or comparisons described in

public educational research institutes are larger them, the results would have been different.than those from other sources. There were no Second, it is important to keep in mind thatother overall study feature relationships, yet in the studies used in this analysis were located bya number of subset analyses study features did searching titles of articles for such words ascorrelate with achievement. Among elementary "programmed learning," "individualized instruc-

school students the most positive results of tion," and "computer-based teaching." We thustechnology have been reported in the most used a common language approach to the

recent studies. It seems likely that technology definition of technology instruction. Technology

instruction may have been used more discrim- instruction was what educational evaluators said

inatingly in recent years. No longer seen as a it was. It turned out that educational evaluators

panacea, technology is now used where it can used the term almost exclusively to describe

contribute the most. The art of developing some type of "packaged" learning experience.

technology materials may have also improved, The occasional class film, televised science special

so that recent studies involve better materials report, or extra materials given by a teacher to

than older studies did. a student are not represented in this meta-

A variable that predicted study outcome for analysis.students using VBI, and particularly with middle Finally, meta-analysis provides a way of

school students, was control for instructor determining major themes in research studieseffect. In studies in which different teachers that have already been done. In this meta-taught VBI and CI classes, examination differ- analysis we looked at the student outcomes thatences were more clearcut and more in favor of were frequently studied: e.g. retention andVBI. In studies in which a single teacher taught achievement. We could not examine less direct,

both classes, differences were less pronounced. subtle or unique outcomes of technology. We

It seems possible that involvement of teachers do not know, for example, whether TI helpedin innovative approaches to instruction may students develop a sense of confidence or self-

have a general effect on the quality of their direction regarding their studies, whether it

teaching. Better lesson construction and prep- contributed to faculty development, or whether

aration of materials may carry over and help it provided the groundwork for future educa-

a teacher to do a better job in their conventional tional innovations.teaching assignments.

The research we have synthesized here is of

We found that more often than not, less the past, but it may have practical implicationsrigorous experimental design affected technology for the future. Regarding both the type of tech-

student achievement in a positive way. In study nology and the student population, the shape

subsets-e.g. low-aptitude, multi-media, middle of the educational technology field in Japan is

school, etc.-using objective criterion tests, changing even today (Akiyama et al., 1981;

equivalent comparison groups and commercially Sakamoto, 1981). Overall our findings on

available criterion tests, examination differences Japanese TI outcomes are in accord with otherwere less clearcut and more favorable to CI; meta-analysis on technology learning done inwhile using non-objective tests, non-equivalent the U.S.-technology instruction stimulates stu-groups, and experimenter-made criterion tests dent achievement and is a legitimate channel forwere more favorable to TI groups. These results conveying educational content.all seem to be in line with those reported byother meta-analysts.

Author notes. The research reported was funded

Three points should be kept in mind by by the Japanese Ministry of Education. All findings,

readers forming an overall evaluation of tech-opinions and conclusions are those of the authorsand do not necessarily represent the views of the

nology teaching on the basis of this meta- Ministry of Education.analysis. The first point concerns the weighting

A complete list of studies used in the analyses

Educational Technology in the Japanese Schools

29

described in this report is available from the first

designated research school (ET). Takamatsu: Shi-author. Address reprint requests to: Barbara J.

gumo Junior High School.Shwalb, Center for Research on Learning and Teach- KAGAWA UNIVERSITY CAI RESEARCH GROUP (1970)ing, University of Michigan, 109 East Madison, Ann

The effectiveness of classroom learning based on aArbor, MI 48109, U.S.A.

CAI system for group use. Educ. Technol. Res.,6(4): 46-69.

KITAKYUSHU MUNICIPAL EDUCATIONAL RESEARCHREFERENCES

INSTITUTE (1975) A study on the systematization ofteaching: Research on teaching methods using

AKIYAMA, T., IMAIZUMI, S., and INOUE, M. (1981)

educational technology (Bulletin No. 7). Kitakyu-Broadcasting and school education in Japan.

shu: Municipal Educational Research Institute,Educ. Technol. Res., 5(2): 15-40.

1-21.ARIYOSHi, K. (1972) Systematization of classes and KONNO, M. et al. (1973) Research on learning and

evaluation of learning (22nd Annual Report). instruction utilizing educational technology (CenterTokyo: All Japan Federation of Educational Re- Bulletin No. 68). Morioka: Iwate Prefecturalsearch Centers, 88-97.

Education Center, 1-50.AzuMA, H. (1979) Educational technology and dis- KULIK, C.-L. C., KULIK, J. A. and BANGERT-DROWNS,

ciplined research: A comment on Professor Fujita's R. L. (1984, April) Effects of computer basedpaper. Educ. Technol. Res., 3: 1-3.

education on elementary school pupils. PaperBABA, B. and YosHIMURA, C. (1982) Experimental

presented at the meetings of the American Educa-analyses of teaching/learning processes (1) Com-

tional Research Association, New Orleans, LA.parisons between conventional method, Piagetian KULIK, C.-L. C., KULIK, J. A. and SHWALB, B. J.method and programmed learning in teaching cal-

(1984) Effects of computer-based instruction withculation of fractions (Bulletin No. 31). Ibaraki:

adults. Unpublished manuscript, University ofIbaraki University, Faculty of Education, 157-

Michigan, Center for Research on Learning and168.

Teaching, Ann Arbor, MI.BRACHT, G. and GLASS, G. (1968) The external KULIK, C.-L. C., SHWALB, B. J. and KULIK, J. A.

validity of experiments. Am. Educ. Res. J., 5: (1982) Programmed instruction in secondary437-474.

school: A meta-analysis of evaluative findings. J.BURNS, P. K. and BOZEMAN, W. C. (1981) Computer-

Educ. Res., 75(3): 133-138.assisted instruction and mathematics achievement:

KULIK, J. A., BANGERT, R. L. and WILLIAMS, G. W.Is there a relationship? Educ. Technol., 21: 31-39.

(1983) Effects of computer-based teaching onCAMPBELL, D. and STANLEY, J. (1963) Experimental

secondary School students. J. Educ. Psychol., 53:and quasi-experimental research designs for re-

143-158.search on teaching. In: N. Gage (Ed.) Handbook KULIK, J. A., COHEN, P. A. and EBELING, B. J. (1980)of research on teaching. Chicago, IL: Rand- Effectiveness of programmed instruction in higherMcNally.

education: A meta-analysis of findings. Educ.CHIBA PREFECTURAL EDUCATIONAL RESEARCH CENTER

Eval. Policy Anal., 2(6): 51-64.(1974) The systematization of learning in Japanese KULIK, J. A., KuLUC, C.-L. C. and COHEN, P. A.language (Bulletin No. 1.31). Chiba: Chiba Prefec-

(1979) Research on audiotutorial instruction: Atural Education Center, 1-27.

meta-analysis of comparative studies. Res. HigherCOHEN, J. (1977) Statistical power analysis for the

Educ., 11: 321-341.behavioral sciences (Revised Edition). New York:

KULIK, J. A., KULIK, C.-L. C. and COHEN, P. A.Academic Press.

(1980) Effectiveness of computer-based collegeCOHEN, P. A., EBELING, B. J. and KULIK, J. A. (1981)

teaching: A meta-analysis of findings. Rev. Educ.A meta-analysis of outcome studies of visual-based

Res., 50: 525-544.instruction. Educ. Commun. Technol., 29(l): 26- KYOTO MUNICIPAL EDUCATIONAL RESEARCH INSTI-36.

TUTE (1971) A study on the systematization ofFucxu, H. and KOTAKI, Y. (1974) Learning guidance learning guidance using educational technology (1)

aimed at the individualization of mathematics: The An experimental study on the effects of educationaldevelopment of learning materials using a learning technology on learning (Research Report No.center (Bulletin No. 7). Aichi: Togo Prefectural 172). Kyoto: Municipal Educational ResearchHigh School, 155-J60.

Institute, 1-75.GLASS, G. (1976) Primary, secondary and meta-

MATSUO, S. (1971) The use of educational technologyanalysis of research. Educ. Res., 5: 3-8.

in social studies instruction. In M. ShuharaHARTLEY, S. S. (1977) Meta-analysis of the effects of (Ed.), The use of educational technology instru-

individually paced instruction in mathematics. Diss. ments in social studies class (pp. 36-98). Tokyo:Abstr. Int., 38(7-A): 4003 (University Microfilms

Meiji Tosho.No. 77-29, 926).

NoGUCHI, M. (1973) The use ofprogrammed learningHIRAYAMA, N. and SATOH, S. (1975) Research on for "schedule development" in junior high school

instruction of probability concepts in mathematics (Educational Science Research Report No. 20).(Center Bulletin for 1974). Morioka: Iwate Prefec- Nagasaki: University of Nagasaki, Faculty oftural Educational Center, 47-65.

Education, 123-132.KAGAWA PREFECTURAL SHIGUMO JUNIOR HIGH OWEN, D. B. (1965) Handbook of statistical tables.

SCHOOL (1971) 1969-70 JHS education curriculum

Reading, MA: Addison-Wesley.

30

B. J. SHWALB et al.

SAKAMOTO, T. (1981) The current state and problems M., AsAI, M. and SuzuKI, H. (1977) A demonstra-

of teaching methods at universities and colleges tion ofthe effects of the use of educational technologyin Japan. Innovations in Higher Education (Report (Research Bulletin No. 9). Aichi: Togo Prefecturalof the Hiroshima/OECD Meeting), 87-109.

High !goof, 57-64.SHWALB, B. J. and SHWALB, D. W. (1984) Instruc- YAsuHIKo, T. and SAKURAI, Y. (1977) A basic ex-

tional technology in adult education: A meta-analysis perimental study of curriculum evaluation in juniorof findings. Unpublished manuscript. University of high school social studies (Research Reports inMichigan, Center for Research on Learning and Educational Science No. 26). Aichi: Aichi Univer-Teaching, Ann Arbor, MI.

sity of Education, 39-59.WAKAI, M., HARADA, A., SHIBATA, I., KOBAYASHI,