Altering the modality of instructions to facilitate imagination:

Interactions between the modality and imagination effects

SHARON TINDALL-FORD & JOHN SWELLER*School of Education, University of New South Wales, Sydney, NSW, 2052, Australia

(*Author for correspondence, e-mail: j.sweller@unsw.edu.au)

Received: 31 August 2004; in final form: 8 December 2005; accepted: 21 December 2005

Abstract. Under some conditions, learning is improved by using a dual mode presen-

tation involving for example, visual diagrams and auditory, rather than written text(modality effect). Under other conditions, learning is improved by asking learners toimagine rather than study instructional material (imagination effect). Both effects have

been explained using cognitive load theory. This paper investigates interactions betweenthe modality and imagination effects. It was hypothesized that the imagination effectwould be facilitated when accompanied by audio/visual instructions compared to visualonly instructions. Experiment 1 provided evidence to suggest that for the materials

used, audio/visual instructions were required to obtain an imagination effect. Experi-ment 2 through verbal protocols aimed to investigate the cognitive mechanismsrequired when studying and imagining and found that learners who studied tended to

engage in search while learners who imagined focused on entities and relations thatneeded to be learned.

Keywords: coginitive load theory, imagination effect, modality effect

This paper explores interactions between two instructional effects,the modality and imagination effects. The modality effect occurswhen instructional material that is presented in dual, audio/visualform is superior to a visual only presentation (e.g. Tindall-Ford etal., 1997). The imagination effect occurs when learners who areasked to ‘‘imagine’’ a procedure or concept learn more than learn-ers who are simply asked to ‘‘study’’ the same procedure or con-cept (e.g. Cooper et al., 2001). In this paper we test the hypothesisthat the imagination effect is more likely to be obtained followingdual mode instruction rather than visual only instruction. Botheffects are based on cognitive load theory (Sweller et al., 1998;Sweller, 1999, 2005; Clark et al., 2006) and it is this theory that isused to predict the interaction between the modality and imagina-tion effects.

Instructional Science (2006) 34: 343–365 � Springer 2006

DOI 10.1007/s11251-005-6075-5

Cognitive load theory is designed to apply some characteristicsof human cognition to instructional design. Sweller (2003, 2004)provided a foundation for cognitive load theory by employing an evo-lutionary view of human cognition designed to indicate why our cog-nitive architecture has some of the characteristics particularly relevantto instruction. Human cognitive architecture includes a working mem-ory that is very limited with respect to both capacity (Miller, 1956)and duration (Peterson & Peterson, 1959) when dealing with novelinformation, but with no known limits when dealing with familiarinformation that has been stored in long-term memory (Ericsson &Kintsch, 1995). Sweller (2003) has suggested that working memoryhas these characteristics because when dealing with new information,there is no plausible central executive available to determine howelements of information should be combined during learning. In con-trast, as indicated below, when dealing with familiar information,schemas held in long-term memory can act as a central executive. If,as occurs when dealing with novel information, there is no centralexecutive to organize elements, they must be organized randomly withthe consequences of any random organization tested for effectivenessusing a problem solving process. For example, failing knowledge,problem solving moves must be determined randomly and tested foreffectiveness. If a person does not know how to get from A to B, theywill have to randomly test particular steps to see if they move closerto the goal. Such a system can only operate when dealing with a verysmall number of elements to prevent combinatorial explosions andhence, when dealing with novel information, working memory mustbe limited.

In contrast, the characteristics of working memory are altered dra-matically when dealing with familiar information organized schemati-cally in long-term memory. Schemas indicate how elements should beorganized, obviating the need for random organization followed bytests of effectiveness (Sweller, 2003). In effect, schemas act as a centralexecutive. As a consequence, when dealing with familiar information,working memory limitations are no longer required and indeed, aresubstantially eliminated (Ericsson & Kintsch, 1995). Unlike whendealing with novel information, schemas, held in long-term memory,can act as a central executive organizing information and actions.There need be no limits to the amount of familiar, schematicallyorganized information transferred from long-term memory that canbe processed by working memory. If a person knows how to get fromA to B, the entire route, that can consist of a huge number of

344

potential choices, can be processed effortlessly in working memory be-cause a schema for the route can be retrieved from long-term memory.In effect, that schema acts as a learned central executive indicatingwhat actions should be taken and when they should be taken. In theprocess, the limitations of working memory are eliminated.

These characteristics of human cognitive architecture have instruc-tional implications (Sweller, 2003, 2004). One of the functions ofinstruction should be to provide a non-random organization of novelinformation. Instruction can act as a substitute for the missing centralexecutive when dealing with novel information. As indicated above,once schemas have been constructed, they can take over the centralexecutive function and instruction is no longer required. Until thatpoint is reached, without suitable instructional material and proce-dures, problem solving is unavoidable and problem solving mustinclude random combinations of elements with tests of effectiveness.

Cognitive load theory has provided a variety of instructional de-sign procedures intended to reduce the need to randomly test elementcombinations while reducing working memory load and so assistingschema acquisition. Instructional design procedures based on cogni-tive load theory derive from a series of effects with each effect gener-ated by the superiority of a particular instructional procedure using acontrolled experimental design. The imagination effect is one example(Cooper et al., 2001; Ginns et al., 2003; Leahy & Sweller, 2005). Theeffect occurs when students demonstrate greater learning and under-standing after they are encouraged to imagine or mentally practice aprocedure that has been demonstrated in a worked example, com-pared to studying the same worked example. Imagining in this con-text requires learners to mentally work through the steps to solutionof a worked example without looking at the example while studyingmeans looking at the example and emphasizing understanding andlearning the relevant procedures and concepts.

Evidence for the effectiveness of imagining (or synonymously,mental practice) on learning has been demonstrated for many yearsover a wide range of physical activities (Surburg, 1968; Schick, 1970;Rawlings & Rawlings, 1974; Romero & Silvestri, 1990; Etnier &Landers, 1996). Much less emphasis has been placed on cognitivetasks but Driskell et al. (1994), conducting a meta-analysis on the effi-cacy of mental practice, found that the greater the cognitive demandsinvolved in a task, the more beneficial mental practice was in increas-ing performance. Cooper et al. (2001), Ginns et al. (2003), and Leahy& Sweller (2005) found that imagining a cognitive procedure was

345

superior to studying the same procedure but only if learners had suffi-cient schematic knowledge to process the material in working mem-ory. For novices with insufficient knowledge, studying was superior toimagining, providing an example of the expertise reversal effect (Kaly-uga et al., 2003).

If the imagination effect occurs only under conditions where learnershave sufficient knowledge to be able to manipulate the relevant materialin working memory, the effect also might be increased under other con-ditions designed to facilitate learning. Reducing working memory loador facilitating working memory functioning may enhance learning.Another cognitive load effect, the modality effect, could provide such aset of circumstances (Low & Sweller, 2005; Mayer, 2005). Contempo-rary research suggests that working memory is not a simple, solitarystructure but composed of multiple channels or processors (Penney,1989; Baddeley, 1992; see also Paivio, 1986). These processors include avisual system for processing visual images and an auditory system fordealing with verbal information. The two systems appear to processtheir different forms of information with some degree of independencewhich means that usable working memory capacity may be increased ifboth systems are used (e.g. visual and auditory processors) rather thanonly one processor. Using Baddeley’s terminology, by presenting verbalmaterial in auditory rather than written form, only the phonologicalloop is required while presenting the material in visual form will requirethe use of both the phonological loop and the visual-spatial sketchpad.Thus, the use of spoken rather than written material should reduce theworking memory load on the visual-spatial sketchpad increasing usableworking memory.

The modality effect occurs under split-attention conditions(e.g. Sweller et al., 1990). Assume multiple visual sources of informa-tion such as a diagram and its associated, written text that, becausethey refer to each other, cannot be understood in isolation. In order tobe understood, visual attention must be split between the multiple sour-ces of information and they must be mentally integrated. Alternatively,they can be physically integrated which reduces cognitive load leadingto the split-attention effect (e.g. Sweller et al., 1990). The use of dualmode presentation acts as a substitute for physical integration. Textualmaterial, rather than being presented in written (visual) form can bepresented in spoken (auditory) form. If, as suggested above, the use ofboth auditory and visual processors increases usable working memorycapacity (see Penney, 1989 for a review), then switching from written tospoken text under split-attention conditions, should facilitate learning.

346

This modality effect has been obtained on many occasions. Researchon the modality effect and instructional design has demonstrated thatstudying instructional materials which employ a dual format consistingof, for example, visual diagrams and auditory text may result in supe-rior learning compared to studying an equivalent visual only presenta-tion consisting of visual diagrams and visual text (e.g. Mousavi et al.,1995; Tindall-Ford et al., 1997; Moreno & Mayer, 1998; Kalyuga et al.,2000; Brunken et al., 2002, 2004; Mayer & Moreno, 2003; Tabberset al., 2004).

In summary, it is suggested the modality effect occurs for thefollowing reasons: (a) Dual modality presentation expands usableworking memory, permitting the same information presented in dualrather than single modality form to be processed more readily. (b)Expanding usable working memory reduces cognitive load which isdetermined by the difference between the working memory resourcesneeded to process information and the working memory resourcesavailable. Cognitive load is high if almost all of the resources availableare needed to process essential information but low if there are excessresources available. (c) If cognitive load is reduced by increasing theworking memory resources available, more resources become availablefor schema construction, the ultimate aim of most instruction.

The current work tested the hypothesis of an interaction betweenthe imagination and modality effects. Experiment 1 compared fourdistinct instructional formats:

1. Audio/visual instructions followed by an imagination component;2. Audio/visual instructions followed with a conventional study based

strategy;3. Visual only instructions followed by an imagination component;4. Visual only instructions followed by a conventional study based

strategy.

It was hypothesized that the imagination effect would be more likelyto be obtained following audio/visual than visual alone presentationtechniques. The first experiment tested this hypothesis while thesecond used verbal protocols to obtain information concerning thecognitive processes occurring under the various conditions.

Experiment 1

The first experiment tested the hypothesis that the imagination effectwas more likely to be obtained under audio/visual than visual only

347

conditions. The specific hypotheses were that the imagination effectwas more likely to be obtained comparing the imagination and studygroups under audio/visual conditions than visual only conditions be-cause more working memory capacity should be available to learnersto imagine the information under the audio/visual conditions. Underthe visual only conditions, if insufficient working memory capacity isavailable to imagine the information, the imagination effect may beinhibited.

The curriculum area used was a sub-section of a junior high schoolmathematics curriculum. Learners were taught how to construct a fre-quency table, how to sum frequencies and how to calculate the mean,mode and range using the table.

Method

ParticipantsForty-four male eighth grade students from a Sydney private second-ary school participated in this experiment. Students were of similarsocio-economic (middle-class) and ethnic (Caucasian) background andwere turning 14 years of age during the school year. The school hadsix eighth grade classes. Students were organized into three levels ofmathematics based on their end of 7th grade mathematics exam re-sults. The top class consisted of students who were acceleratedthrough the mathematics curriculum and the sixth class consisted ofstudents who were receiving remedial instruction in mathematics. Forthe purpose of this experiment, only students from the four middleability classes were tested. These students were chosen because pilottesting indicated that the materials used in the experiment were suit-able for these students but too easy for many students in the top classand too difficult for students in remedial classes. All students hadcompleted the algebra unit of the eighth grade syllabus but had notbeen introduced to basic statistics, which was the instructional mate-rial used in this experiment.

Materials and procedureParticipants were randomly allocated to one of the four groups withthe only restriction being that, as far as possible, equal numbers ofparticipants from each class were allocated to each group. The visualonly instructions presented to the two Visual-only groups consisted ofa frequency distribution table and related textual statements. Figure 1provides an example of visual-only study instructions. Neither thetable nor the text was intelligible as separate entities. An understanding

348

of the instructions and how to perform a particular statistical opera-tion could only be made by mentally integrating the text with therelated diagrammatic entities. The audio/visual instructions providedthe same frequency distribution table and textual information, how-ever the textual instructions were provided in an auditory form viaa tape recorder. The length of the audio instructions determinedthe time allocated to either reading the instructions or listening tothe tape.

Figure 1. Instructional material used in section 1 of both experiments.

349

There were eleven participants in each group and each partici-pant was tested individually. There were four instructional sectionsdemonstrating basic statistical concepts and operations. Each sec-tion increased the complexity of information presented and reliedon an understanding of the previous sections. Each section incorpo-rated a learning phase consisting of reading or listening to theinstructional material, a practice phase consisting of studying orimagining the instructions, and a test phase incorporating testquestions based on what had been previously learned.

The section 1 learning phase had a time limit of 70 s to read or lis-ten to the taped instructions. The instructions demonstrated how datacould be organized into a frequency distribution table. These instruc-tions were followed immediately by a practice (60 s) and test phase(120 s) for this section. The section 1 test phase required students toenter data into a frequency distribution table. The data was presentedto students in raw form and they had to record each score value andthe frequency of each score. Three test marks were allocated to thistest section. Figure 1 demonstrates the instructional material used insection 1.

The section 2 learning phase instructions (110 s) demonstratedhow, on a frequency distribution table, the score value (x) may bemultiplied by the frequency value (f), to calculate the fx value. Theseinstructions were immediately followed by a practice (90 s) and test(120 s) phase for this material. This test phase required students to fillin the missing numbers on a frequency distribution table. Studentshad to calculate the fx value or if presented with the fx value calcu-late either the score or frequency value that was missing. Twelvemarks were allocated to this test section.

Section 3 (110 s) showed how to calculate the mean from a fre-quency distribution table by calculating the sum of all scores, thendividing by the sum of the frequency value. A practice (90 s) and thena test (300 s) phase followed based on the instructional material pre-sented in this section. The two test questions for section 3 requiredstudents to fill in the missing data on the frequency distribution table,calculate the fx column, calculate the sum of the frequency columnand the sum of the fx column and use this information to calculatethe mean. Fourteen marks were allocated to this test section. Figure 2shows the visual only instructional material used for section 3 forboth the imagination and study groups.

The section 4 learning phase instructions (90 s) demonstrated howto calculate the mean, range and mode from a frequency distribution

350

table. These instructions too were immediately followed by a practice(120 s) and test (600 s) phase. The first two test questions for thissection required students to calculate the mean, mode and range from

Figure 2. Instructional material used in section 3 of both experiments.

351

a frequency distribution table, while the last question required stu-dents to initially organize the data into a frequency distribution tableand then calculate the range, mode and mean. Twenty nine markswere allocated to this test section.

As indicated above, a practice phase immediately followed eachlearning phase for each of the four sections. The practice phase was acontinuation of the learning phase. Participants were asked to eitherstudy or imagine the procedures they had just been learning. Theinstructions provided to both the study and imagination groups wereidentical to the information shown during the learning phase exceptthat the spoken information was no longer available to the audio-visual groups and the identical written information was eliminated forthe visual only groups. Thus, only the tabular/diagrammatic informa-tion of a frequency table without explanatory information wasavailable during the practice phases of each of the four sections.

Prior to students starting Experiment 1, each participant was givena verbal overview describing what was required during the practicephases when either studying or imagining. Students in the Studygroups were instructed as follows:

When studying, try and understand the information shown, tryand make sense of the instructional material and what statisticalprocedure is being demonstrated. Attempt to understand the stepsto solve the problem. Read through the information carefully andexamine the diagram.

Students in the Imagination groups were told:

When imagining, try to imagine yourself working out the problem,close your eyes or look away from the information and thinkabout yourself actually carrying out the steps and solving theproblem. Attempt to imagine the steps needed to solve theproblem. If you forget how to do the problem look back at theinstructional material.

Students were then provided with an example unrelated to theexperimental materials to study or imagine. The example showed arectangle with measurements for its length and breadth. Under thediagram was the mathematical solution for calculating the perimeterof the rectangle. Students in the Study group were told to ‘‘study theinformation, read through the text, study the diagram and try andunderstand the information’’. The Imagination groups were providedwith exactly the same example of a rectangle with the corresponding

352

mathematical calculations. Students in the Imagination groups weretold when imagining the information ‘‘turn away from the materialand try and imagine yourself actually doing the problem, that is, per-forming the steps to solve the problem.’’ This introduction to thestrategies of either studying or imagining was used to try to ensurethat students had an understanding of and employed these strategiesduring each practice phase. After each set of learning materials,students were given a specified practice time to study or imagine.

As indicated above, test phases followed the learning and the prac-tice phases for each of the four sections. In all test sections, once astudent completed a question the student could not return to thequestion nor use it as a reference for future questions. All test ques-tions were objective and aimed to assess the students’ ability to applythe statistical processes and procedures that had been introducedduring the learning and practice phase. No test questions requiredreplication of what was previously shown in the learning and acquisi-tion phases. An example of the test material for section 2 is shown inFigure 3. It should be noted that while each test question had a maxi-mum time, learners could finish earlier. The time to find a solution

Figure 3. Example of test materials used in Experiment 1, section 2 part 2.

353

was noted, with learners who failed to complete a question within theavailable time allocated the maximum time. While time to solutionwas not emphasised to participants, it should be noted that all partici-pants were aware that their time to solution was being recorded.

Results and discussion

The variables under analysis were scores on test items. Means and stan-dard deviations are displayed in Table 1. A 2 (visual vs. audio-visual) �2 (imagine vs. study) analysis of variance was completed on overall testscores. There was a significant audio/visual effect, F(1, 40)=5.34,MSe=78.80, p=0.03, Cohen’s d=0.7, a significant imagination effect,F(1, 40)=7.95, MSe=78.80, p=0.01, d=0.85 and a significant audio/visual by imagination interaction, F(1, 40)=5.82, MSe=78.80, p=0.02,d=0.73.

Following the significant interaction, simple effects testing indi-cated a significant difference between the audio/visual-imagine andaudio/visual study groups, F(1, 20)=12.95, MSe=82.19, p=0.002,d=1.53, but no significant difference between the visual imagine andvisual study groups, F(1, 20)=0.09, MSe=75.93, p=0.77, indicatingthe imagination effect was only obtainable under audio/visual condi-tions. Clearly the significant interaction is due primarily to the superi-ority of the audio/visual imagination group. To investigate if theaudio/visual and imaginations effect were more robust on more com-plex information, remembering each section became more complexand introduced more concepts, follow up ANOVA’s of the four indi-vidual section tests were conducted. Analyses indicated an identicalpattern of significance to the over all test.

The results demonstrate that the imagination effect was onlyobtainable under audio/visual but not under visual only conditions.We argue that the audio/visual technique facilitated schema construc-tion, with imagination then allowing schema automation. Students’ability to imagine the information was enhanced when audio/visual

Table 1. Means and standard deviations for test scores in Experiment 1; n=11

Audio/visual

Imagination

Audio/visual

Study

Visual

Imagination

Visual

Study

Mean 50.82 36.82 38.18 37.09

SD 5.90 11.34 7.03 10.12

354

instructions were used prior to imagining the information. To imaginea mathematical procedure, it must be possible for all elements of thatprocedure to be processed in working memory. A schema incorporat-ing the relevant elements assists in this process and the audio/visualtechnique facilitated schema construction. Schema construction, inturn, permits multiple elements of information to be treated as a sin-gle element in working memory. (A schema for the word ‘‘cat’’, forexample, permits competent readers to treat the word as a single ele-ment in working memory rather than as three separate letters.) Onceschema construction freed sufficient working memory resources to al-low the effective imagination of the statistical procedures required,automation could commence. In contrast, study techniques followingaudio/visual instruction did not make as much use of the newly ac-quired schemas while visual only instruction did not even permit suffi-cient schema construction to commence, rendering subsequentimagination instructions ineffective.

With respect to the modality effect, it might be noted that whileaudio/visual instructions were superior to visual only instructionsunder imagination conditions contributing to the significant interac-tion, there was no modality effect under study conditions. A possibleexplanation is that the tests failed to detect differences in schemaacquisition until the schemas became automated under imaginationconditions. Schemas may have been better acquired under audio/visualconditions but may not have been usable during the tests until theyhad become automated.

Experiment 1 indicated the efficacy of an imagination instructionalstrategy when combined with audio-visual instructions. The next exper-iment aimed to further investigate the effectiveness of audio/visual andimagination instructions via the use of verbal protocols in order toascertain the procedures used by students when they were required toimagine or study.

Experiment 2

Experiment 1 provided quantitative evidence that students whoengaged in imagining, rather that studying information may havea greater understanding of statistical concepts and procedures.However, the superiority of imagining was only found following audio/visual instructions. The aim of Experiment 2 was to further explore thisphenomenon, by qualitative rather than quantitative analysis. Theobjective was to provide information concerning the cognitive

355

processes involved in imagination by collecting verbal protocols duringthe practice phase in Experiment 2. As was the case in Experiment 1,Experiment 2 included learning, practice, and test phases. During thelearning phase, students were provided learning materials in audio-vi-sual or visual only format, followed by a practice phase using the samematerials without the auditory component or the equivalent writtencomponent for the visual only groups, where students either studied orimagined the statistical processes.

The verbal protocols of Experiment 2 were intended to provideevidence for two hypotheses. (1) It was hypothesized that learnersasked to study the materials would engage in a search for whatneeded to be learned while learners asked to imagine would directlyengage in the learning process. Search requires working memoryresources and interferes with learning (e.g., Sweller & Cooper, 1985;Cooper & Sweller, 1987). (2) It was hypothesized that the verbalprotocols would provide evidence that imagining is assisted when pre-ceded by audio/visual instructions compared to visual only instruc-tions because of enhanced learning under audio/visual instructions.

A number of techniques were employed in this experiment to maxi-mize the validity of the verbal reports. Based on the work of Ericssonand Simon (1993) concerning procedures that increase the validity ofverbal protocols, the following strategies were used. An unaided ver-balization procedure was adopted where the students were asked toprovide a verbal monologue while either imagining or studying. Theverbal monologue was audio taped and later transcribed. Minimumintervention by the researcher occurred, but when a student pausedfor a period of time, the researcher would ask the student to ‘‘tell mewhat you are thinking about’’. Students were instructed not to tryand explain their thoughts, but merely to narrate their thoughts.

Method

ParticipantsSixteen female students from a Sydney girls’ secondary school, partic-ipated in Experiment 2. (It should be noted that past research onboth the modality and imagination effects has not obtained anydistinctions between males and females – see Mousavi et al., 1995;Cooper et al., 2001; Ginns et al., 2003.) Students were of similarsocio-economic (middle class) and – ethnic (Caucasian) background.All students were turning 13 years of age during the school year andwere presently completing seventh grade. The school had five seventhgrade classes. Only students from the top mathematics class took

356

part in the experiment. In terms of knowledge and ability, pilotstudies indicated this class was most similar to the students used inExperiment 1. The top class was formed based on results obtainedfrom a series of mathematics examinations that students completed atthe beginning of the school year. The pilot studies indicated that stu-dents in the middle and lower ability classes may have had difficultylearning the materials, as they had not been taught sufficient algebrato understand the statistical concepts being introduced. Unlike Exper-iment 1, which was conducted at the beginning of the school yearwith middle ability students from Year 8, this experiment was con-ducted near the end of the year with higher ability students in Year 7.Thus, there was about a 3 month difference in amount of schoolingreceived by the participants between the two experiments.

Materials and procedureThe instructions and procedures used in Experiment 2 were similar tothose used in the previous experiment. As was the case for Experi-ment 1, a 2� 2 experimental design was used. Because the intentionwas to collect verbal protocols rather than analyze quantitative data,there were only 4 participants per group.

Modifications were made to reduce the time required to test eachparticipant because of school stipulations and the added time requiredto collect verbal protocols. For this reason, participants in this experi-ment did not complete section 1 or section 2 test questions. Studentsin Experiment 2 completed the following;

1. Section 1 Learning and Practice Phase,2. Section 2 Learning and Practice Phase,3. Section 3 Learning, Practice and Test Phase,4. Section 4 Learning, Practice and Test Phase.

The description of instructional, practice material, times and marksallocated for each section can be found in the description ofExperiment 1.

During the imagination/study component, students were told to tryand keep up a running commentary of what they were doing, includ-ing their thought processes. Participants in the imagination groupswere told that they should tell the experimenter ‘‘what you are imag-ining’’. Students from the study group were told they should indicate‘‘what you are thinking about or looking at’’. During the study orimagination process, if students failed to provide a commentary, theexperimenter prompted them with the above statements. A test phasefollowed the learning and practice phase. Students completed test

357

questions from Sections 3 and 4 of Experiment 1. All questions andmarking were identical to Experiment 1 (see Table 1).

The following procedures were used to analyse the verbal protocolsand provide evidence concerning the two hypotheses. We hypothe-sized that learners asked to study the materials would engage in asearch for what needed to be learned while learners asked to imaginewould directly engage in imagining the procedures. We looked forevidence of this hypothesis by comparing the extent to which searchstatements such as ‘‘looking for’’, ‘‘searching’’, ‘‘attempting to findout’’, ‘‘checking’’ or equivalent terms were characteristically used bythe study but not imagination groups while direct procedural state-ments such as ‘‘I am doing’’, ‘‘I am calculating’’, ‘‘I am workingthrough’’ were used by the imagination but not the study groups.

The hypothesis that imagining is assisted when preceded by audio/visual instructions compared to visual only instructions because ofenhanced learning under audio/visual instructions was assessed byconsidering the positive and negative imagery statements made byaudio/visual and visual only groups. A positive imagery statement isone that indicates an ability to imagine while a negative imagerystatement is one that indicates difficulty in imagining.

Results and discussion

As stated previously, the purpose of this study was to obtain qualita-tive data through detailed verbal protocols that were collected duringthe practice phase when students had to either imagine or studythe information. Participants were encouraged to provide a runningcommentary in order to give an insight into their cognitive processes.This commentary was recorded and then later transcribed andanalyzed.

Cognitive processes used by learners in the imagination and studygroupsAnalysis of students’ comments from the two study groups, indicatethat searching is a common element in many of the verbal protocols.Search statements are defined as ones in which learners includeterms such as ‘‘looking for’’, ‘‘searching’’, ‘‘attempting to find out’’,‘‘checking’’ or equivalent terms. Search appears to be one of the keyprocesses involved when asked to study as can be seen from the fol-lowing examples: ‘‘I am looking at the answer to find information’’,‘‘I am looking for all possible answers....just looking for number

358

patterns’’ and ‘‘I kind of figure out all the kinds of ways I could fig-ure this out’’ and ‘‘I keep searching making up my own examples’’.

Research has repeatedly demonstrated the negative effect of thesearch process for learning and understanding (e.g., Sweller & Cooper,1985; Cooper & Sweller, 1987). While these studies compared problemsolving search with studying worked examples, the present work sug-gests that search occurs during studying as well. A search process pla-ces heavy demands on limited working memory resources, leaving fewfor the task of learning. Differential search may provide a valid expla-nation why study groups performed relatively poorly on test questionscompared to imagination groups in Experiment 1. The process ofsearching reduces focus. Statements like, ‘‘I am mostly looking, I’msearching, I am checking’’ and ‘‘I am looking over things’’ confirm thissuggestion. Every study student, regardless whether they were in theaudio/visual study or visual study group made some reference toengaging in some form of search process when studying.

In contrast, the verbal protocols collected from the imaginationgroups suggested when a person effectively imagines a procedure amore focused cognitive engagement appears to occur. Statements like,‘‘I am doing’’, ‘‘I am calculating’’, ‘‘I am working through’’, ‘‘I’mvisualizing’’ or the equivalent were common phrases used by bothimagination groups that indicated focused cognitive processing.Examples suggesting a successful imagination process included; ‘‘I gothrough each number in the score column and multiply it by its fre-quency so one times three is three, then two times its frequency oftwo is four, I am doing each one now and to do the sum of thefrequency I am going through and adding each number together’’.When the imagination process is used correctly, it appears to providesome framework and structure for learning and encourages deliberate,organized practice. The imagination group provided no statementsconcerning search.

Imagination is assisted by the use of audio/visual instructionsVerbal protocols indicated that the imagination process appeared tobe assisted when coupled with audio/visual instructions. Positiveimagery statements were determined by students’ verbal protocolsindicating active cognitive processing. Some examples are; ‘‘I can seemyself doing each one of these steps’’; ‘‘I multiply each one, that iswhat I am doing now’’ and ‘‘what I am doing now is dividing thefx by the f ’’. This can be contrasted to comments from students inthe visual-imagine group who indicated on occasions an inability toimagine a process. This difficulty in imagining was defined by negative

359

statements and is demonstrated by the following verbal protocols;‘‘I don’t know how to imagine it’’; ‘‘I’m not sure, I don’t quiteunderstand’’ and ‘‘I’m trying to find out ...um... it is not clear reallywhat I should do’’. There was only one statement from the audio/vi-sual-imagination group suggesting an inability to imagine.

The visual imagination group tended to make tentative statementsdefined by words such as; ‘‘I might’’, ‘‘I guess’’ or equivalent. Exam-ples are: ‘‘I imagine how it all works out ... I imagine I would gothrough all the numbers from 1 to 10’’ and ‘‘I guess I keep goingthrough, working the numbers’’. The visual-imagination group’sverbal protocols indicate vague, poorly defined thought processes.These thought processes can be compared to statements from theaudio/visual-imagination group which demonstrated a deliberate,focused pattern of thinking; ‘‘I see the table, have a picture of it inmy head, so I can remember it.... multiply the score by the frequencyin my head, which I can do ... then I get the answer’’ and ‘‘what I amdoing is multiplying the score by the f value to get each fx value’’.

Also evident through the analysis of verbal protocols is the frac-tured and on occasions incorrect nature of the visual-imagination stu-dents’ understanding of concepts; ‘‘ first the mode is the most popularnumber ... the mode on the table is five, the most popular number; Iwould then subtract it from the smallest number to get 5’’. This pro-vides an example of a student confusing the mode and range.Although this student could provide the correct meaning for themode, she failed to understand the process of finding the mode fromthe frequency distribution table. She had formed an incorrect proce-dure for calculating the mode and on all test questions requiring thecalculation of the mode performed a subtraction. A comparison maybe made with the audio/visual-imagination group where all four stu-dents made reference to looking at the frequency column to find thelargest number and then observing what number from the score col-umn corresponded with the frequency value; ‘‘The mode, the numberthat occurred the most. I look down the frequency column, find thelargest frequency, I am scanning in my head that column then I lookat the score number next to the largest frequency number’’.

In summary, the verbal protocols indicate some of the cognitiveprocesses that can be used to explain the quantitative results ofExperiment 1. Imagining material results in processing that is focusedon the entities and relations that need to be learned while studyingthe material can result in the use of a somewhat vague search for rele-

360

vant patterns. Furthermore, that process of imagining is assisted bylearning using audio/visual rather than visual only materials.

General discussion

The cognitive processes engaged in by students imagining audio/visualmaterials are complex. Nevertheless, information is becoming availableconcerning many of the relevant variables and relations and thatinformation is beginning to have instructional implications. Cognitiveload theory (Sweller, 2005) with its emphasis on the instructionalimplications of human cognitive architecture can frequently be used asa guide. The theory has previously generated the instructional modal-ity (Tindall-Ford et al., 1997) and imagination (Leahy & Sweller,2005) effects. Particular interaction patterns between these two effectsare suggested by the theory. Those interactions were investigated inthe current work.

Some instructional materials necessarily include disparate sourcesof information that must be mentally integrated before they becomeintelligible because each source of information is unintelligible inisolation (Sweller et al., 1990). Integrating disparate sources of infor-mation imposes a heavy working memory load. Presenting some ofthat information in spoken or auditory mode rather than presentingall of it in written or visual mode can reduce the load on visual work-ing memory. In other words, transferring some of the information tothe auditory channel should reduce the load imposed on the visualchannel. Experiment 1 provided evidence for this modality effect withaudio/visual instructions proving superior to visual only instructionsunder imagination (although not study) conditions.

Instructional materials, which are imagined, can sometimes belearned better than materials that are merely studied. Whether imag-ining or studying is better depends, at least in part, on the levels ofexpertise of the learners. In order to imagine a procedure, the sche-mas associated with it must be sufficiently established to enable thelearner to process the procedure in working memory. Without thatpossibility, imagination instructions may be difficult or impossible tofollow, resulting in improved performance by learners studying ratherthan imagining the material. Thus, an imagination effect may only beobtained using material for which learning has already commenced.In the initial phases of learning, the effect is reversed with studyinstructions proving superior. While not tested in the present

361

experiments, that effect has been consistently obtained (Cooper et al.,2001; Ginns et al., 2003; Leahy & Sweller, 2005).

If the imagination effect requires that learning be advanced tosome extent and if the modality effect indicates that dual modepresentations advance learning more than single mode presentations,it follows that the imagination effect is more likely to be obtainedfollowing dual mode instructions rather than single mode instructions.Experiment 1 demonstrated the imagination effect under audio/visualconditions but not under visual only conditions. In turn, the verbalprotocols of Experiment 2 provided evidence for some of the pro-posed theoretical constructs. They suggested that learners processedthe material differently when asked to study or imagine it and thatthe imagination process was assisted by the use of audio/visualinstruction.

Instructional implications flow from the results of theseexperiments. First, obtaining the modality effect under imaginationconditions and obtaining the imagination effect under audio/visualconditions confirm the importance of these instructional designeffects. When presented with multiple sources of information thatcannot be understood in isolation, using dual mode instructions canbe beneficial. (It must be emphasized, that previous work has indi-cated that these may be the only conditions under which dual modeconditions are beneficial, e.g., Kalyuga et al., 2000.) In addition, ask-ing learners to imagine rather than study information to be learnedalso may be beneficial providing they have progressed sufficiently toenable them to imagine the information. The interaction between themodality and imagination effects provides the major new finding ofthis study. If dual modality presentation can enhance learning (asoccurred under imagination conditions) and if some degree of learn-ing is essential before the material can be imagined, then dual modepresentation followed by imagination instructions should be advanta-geous. That result was obtained.

The current work demonstrated the advantages of using dualmodality instruction when asking learners to imagine rather than sim-ply study examples. Given the failure to find a modality effect understudy conditions, further work is required to delineate the preciseconditions under which this interaction may or may not occur. Forexample, Tindall-Ford et al. (1997) found that the modality effect wasonly obtainable under conditions of high rather than low informa-tional complexity. It is reasonable to suppose that similar results maybe obtainable for the interaction between the modality and imagina-

362

tion effects. Lastly, while the current results were obtained under real-istic conditions, the study (particularly Experiment 1) was conductedas a controlled experiment. Long-term educational studies testing thesuggestion that learning will be facilitated if learners are presenteddual-modality instructions and asked to imagine rather than studyinstructional material are required.

Acknowledgements

The work reported in this paper was supported by a grant from theAustralian Research Council to the second author. The authors wish tothank Brother Patrick Howlett, the Principal of Marcellin College andthe staff and students of Marcellin College for their participation inExperiment 1. We also acknowledge the support and assistance of thePrincipal, Miss Rosalyn Bird and the staff and students of Danebank,Anglican School for Girls Hurstville who participated in Experiment 2.

References

Baddeley, A. (1992). Working memory. Science 255: 556–559.

Brunken, R., Plass, J. & Leutner, D. (2004). Assessment of cognitive load in multimedialearning with dual-task methodology: Auditory load and modality effects.Instructional Science 32: 115–132.

Brunken, R., Steinbacher, S., Plass, J.L. & Leutner, D. (2002). Assessment of cognitiveload in multimedia learning using dual-task methodology. Experimental Psychology49: 1–12.

Clark, R., Nguyen, F. & Sweller, J. (2006). Efficiency in learning: evidence-basedguidelines to manage cognitive load. San Fransisco: Pfeiffer.

Cooper, G. & Sweller, J. (1987). Effects of schema acquisition and rule automation

on mathematical problem solving transfer. Journal of Educational Psychology 79:347–362.

Cooper, G., Tindall-Ford, S., Chandler, P. & Sweller, J. (2001). Learning by imagining.Journal of Experimental Psychology: Applied 7: 68–82.

Driskell, J.E., Copper, C. & Moran, A. (1994). Does mental practice enhanceperformance? Journal of Applied Psychology 7: 481–492.

Ericsson, K.A. & Kintsch, W. (1995). Long-term working memory. Psychological

Review 10: 211–245.Ericsson, K.A. & Simon, H.A. (1993). Protocol analysis: Verbal reports as data.

Cambridge, Mass: MIT Press, Revised edn.

Etnier, J.L. & Landers, D.M. (1996). The influence of procedural variables on theefficacy of mental practice. The Sport Psychologist 10: 48–57.

Ginns, P., Chandler, P. & Sweller, J. (2003). When imagining information is effective.

Contemporary Educational Psychology 28: 229–251.

363

Kalyuga, S., Ayres, P., Chandler, P. & Sweller, J. (2003). The expertise reversal effect.

Educational Psychologist 38: 23–31.Kalyuga, S., Chandler, P. & Sweller, J. (1998). Levels of expertise and instructional

design. Human Factors 40: 1–17.

Kalyuga, S., Chandler, P. & Sweller, J. (2000). Incorporating learner experience into thedesign of multimedia instruction. Journal of Educational Psychology 92: 126–136.

Leahy, W. & Sweller, J. (2005). Interactions among the imagination, expertise reversaland element interactivity effects. Journal of Experimental Psychology: Applied 11:

266–276.Low, R. & Sweller, J. (2005). The modality principle. In R.E. Mayer, ed., Cambridge

handbook of multimedia learning, pp. 147–158. Cambridge University Press: New

York.Mayer, R. (2005). Cognitive theory of multimedia learning. In R.E. Mayer, ed.,

Cambridge handbook of multimedia learning, pp. 31–48. Cambridge University Press:

New York.Mayer, R. & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia

learning. Educational Psychologist 38: 43–52.Miller, G.A. (1956). The magical number of seven, plus or minus two: Some limits on

our capacity for processing information. Psychological Review 63: 81–97.Moreno, R. & Mayer, R.E. (1998). A split-attention effect in multimedia learning:

Evidence for dual-processing systems in working memory. Journal of Educational

Psychology 90: 312–320.Mousavi, S.Y., Low, R. & Sweller, J. (1995). Reducing cognitive load by mixing auditory

and visual presentation modes. Journal of Educational Psychology 87: 319–334.

Paivio, A. (1986). Mental representations: a dual coding approach. Oxford, England:Oxford University Press.

Penney, C.G. (1989). Modality effects and the structure of short term verbal memory.

Memory and Cognition 17: 398–422.Peterson, L.R. & Peterson, M. (1959). Short term retention of individual items. Journal

of Experimental Psychology 58: 193–198.Rawlings, E.I. & Rawlings, I.L. (1974). Rotary pursuit tracking following mental

rehearsal as a function of voluntary control of visual imagery. Perceptual andMotorSkills 38: 302–309.

Romero, K. & Silvestri, L. (1990). The role of mental practice in the acquisition and

performance of motor skills. Journal of Instructional Psychology 17: 218–221.Shick, M. (1970). Effects of mental practice on selected volleyball skills for college

women. Research Quarterly 41: 88–94.

Surburg, P.R. (1968). Audio, visual and audio-visual instruction with mental practice indeveloping the forehand tennis drive. Research Quarterly 39: 728–734.

Sweller, J. (1999). Instructional design in technical areas. Victoria, Australia: ACERPress.

Sweller, J. (2003). Evolution of human cognitive architecture. In B. Ross, ed., Thepsychology of learning and motivation, pp. 215–266. Academic Press: San Diego.

Sweller, J. (2004). Instructional design consequences of an analogy between evolution by

natural selection and human cognitive architecture. Instructional Science 32: 9–31.Sweller, J. (2005). Implications of cognitive load theory for multimedia learning. In

R.E. Mayer, ed., Cambridge handbook of multimedia learning, pp. 19–30. Cambridge

University Press: New York.

364

Sweller, J., Chandler, P., Tierney, P. & Cooper, M. (1990). Cognitive load as a factor in

the structuring of technical material. Journal of Experimental Psychology: General119: 176–192.

Sweller, J. & Cooper, G. (1985). The use of worked examples as a substitute for problem

solving in learning algebra. Cognition and Instruction 2: 58–89.Sweller, J., van Merrienboer, J. & Paas, F. (1998). Cognitive architecture and

instructional design. Educational Psychology Review 10: 251–296.Tabbers, H., Martens, R. & Van Merrienboer, J. (2004). Multimedia instructions and

cognitive load theory: Effects of modality and cueing. British Journal of EducationalPsychology 74: 71–81.

Tindall-Ford, S., Chandler, P. & Sweller, J. (1997). When two sensory modes are better

than one. Journal of Experimental Psychology: Applied 3: 257–287.

365