audiovisual incongruence in multimedia: an exploratory eeg study

AUDIOVISUAL INCONGRUENCE IN MULTIMEDIA: AN EXPLORATORY EEG STUDY

BY

MICHAEL ZWICK

A Thesis

Submitted to the Department of Psychology, Neuroscience, and Behaviour

in Partial Fulfillment of the Requirements

for the Degree Honours Bachelor of Science

McMaster University

April 2013

2


HONOURS BACHELOR OF SCIENCE (2013)

MCMASTER UNIVERSITY

Hamilton, Ontario

TITLE: Audiovisual incongruence in multimedia: an exploratory EEG study

AUTHOR: Michael Zwick

SUPERVISORS: Dr. Joe Kim & Dr. Jen Heisz

NUMBER OF PAGES: 28

3


Abstract:

Despite the prominent use of multimedia in education, very little is known about the

cognitive processes that underlie learning from multimedia. In order to explore this topic, we

used electroencephalography (EEG) to measure subjects' brain wave activity as they viewed

a simulated online lecture consisting of images and narration, and then tested subjects on

their knowledge of the lecture material. Since alpha power has been shown to increase with

task difficulty (Cooper et al. 2002; Galin, Johnstone, Herron, 1978; Handel, Haarmeier,

Jensen, 2011; Jensen, Gelfand, Kounios, Lisman, 2002; Osaka, 1984; Tuladhar et al., 2007),

we examined how it would be altered by changes in lecture design and whether this affects

learning of lecture material. We designed the lecture so that one half was higher in cognitive

load than the other. For one half of the lecture, the images did not directly correspond to the

narration (“incongruent half”), whereas the other half did not contain this audiovisual

incongruence (“congruent half”). Subjects showed greater alpha power while watching the

incongruent half of the lecture than the congruent half. Subjects also performed worse on quiz

questions derived from material presented in the incongruent half. This suggests that

audiovisual incongruence imposes great demands on working memory and is detrimental to

learning. These results provide evidence that EEG allows insight into how students learn from

multimedia and highlight the value of using EEG in pedagogical research.

4


Acknowledgements:

I would like to thank everyone who supported me and helped me with completing my

thesis. In particular, I would like to thank my supervisors Dr. Joe Kim and Dr. Jen Heisz for all

of their valuable advice and guidance, as well as Barbara Fenesi for mentoring me throughout

the year. Special thanks also goes to Ali Hashemi for teaching me how run EEG, Katherine

Lajkosz for her help in conducting the experiment, and Dr. Patrick Bennett and Dr. Allison

Sekuler for use of their lab.

5


Table of Contents

Abstract 3

Acknowledgements 4

Introduction 6-11

Methods 12-15

Results 16

Discussion 17-21

Figures

Figure 1. Topoplot comparing differences in alpha power 22

Figure 2. Mean alpha power exhibited during each half of the lecture 23

Figure 3. Mean comprehension quiz scores by question type and origin 24

References 25-27

Appendix 28

6


Introduction:

Over the past couple decades there has been a clear influx of technology into

education. Especially in higher education, instructors are making increased use of multimedia,

such as PowerPoint presentations, interactive animations, and online lectures, as vehicles to

present course content to students (Apperson, Laws, Scepansky, 2008; Bartsch & Cobern,

2003). Multimedia instruction has been touted as an antidote for student disengagement and

as a way to enrich learning (Beeland, 2002; Jacques, 1995; Cutrim Schmid, 2008). These

claims are supported by research in cognitive psychology and the popularity of instructional

multimedia among students (Najjar, 1996; Shuell & Farber, 2001). Given this, it is likely that

multimedia will continue to have a prominent role in education in the years to come. In

response, the focus of pedagogical research has shifted towards understanding the effects of

incorporating multimedia into teaching practice and how to design instructional media to best

facilitate learning. However, there is comparatively less known in regards to the cognitive

processes that underlie multimedia learning, which was the focus of the present study. We

sought to determine whether a variation in multimedia presentation design results in the

evocation of different cognitive processes and how this affects learning of instructional

material.

When we take into account Baddeley's (1992) model of working memory, the

advantages of multimedia learning become apparent. At the core of this model is the

assumption that visual and auditory information are processed in independent channels within

working memory, referred to as the visuospatial sketchpad and phonological loop,

respectively. These channels are thought to have a finite capacity for the amount of

information they can hold or process at any given time, and while they are distinct, they

7


interact with one another. Multimedia presentations typically include visual components, such

as images or written text, as well as some sort of narration or auditory information, thereby

stimulating both of these channels. Therefore, learners must manipulate these different forms

of sensory stimuli within their working memory to create a cohesive whole and integrate this

with previous knowledge stored in long-term memory. Not only does this promote more

elaborate processing of instructional material, but it also allows for the distribution of cognitive

load among these two channels, so no one channel becomes overburdened.

However, not all educational multimedia is created equal. One way to judge the

effectiveness of a multimedia presentation is to examine the cognitive processes it evokes

and determine to what extent these processes tax limited working memory resources. In other

words, we can measure the cognitive load imposed by instructional media. Although cognitive

load is important to consider when evaluating the effectiveness of a multimedia presentation,

it is difficult to assess, as cognitive processes are hidden from observation. It is for this reason

that cognitive load is often assessed using subjective measures. Studies often make use of

self-reports to measure the difficulty or engagement of multimedia materials. However, this

approach is flawed as perceived understanding does not always accurately reflect actual

understanding (Benjamin & Bjork, 1996; Kornell & Bjork, 2008). Even when objective

measures are used, they typically provide only an indirect assessment of cognitive load. A

common research approach is to have subjects watch one of two multimedia presentations

that have the same content but differ in some aspect of their design, and then quiz subjects

on the content of the presentation (Bruenken, Plass, Leutner, 2003). The logic is that poorer

quiz performance should be an indicator of the amount of cognitive load induced by

instructional design, however learning outcomes can be influenced by quiz design and

8


individual differences in working memory capacity (Mayer, 2001). Behavioural measures also

have another significant disadvantage in that they cannot be used to gauge cognitive load

online during the actual learning process. Therefore, it is difficult to determine how a particular

manipulation in presentation design directly affects cognitive activity and learning, which limits

the types of hypotheses that can be tested. Supplementing behavioural data with a

physiological measure of cognitive load may remove some of these limitations and provide

greater flexibility in the types of questions that can be asked in pedagogical research. This

would allow for a deeper understanding of the processes involved in multimedia learning and

improved testing for the effectiveness of multimedia design.

One promising approach is to borrow physiological measures that are commonly used

in cognitive neuroscience research. By using neuroimaging techniques, such as positron

emission topography (PET), functional magnetic resonance imaging (fMRI), and EEG, we can

determine what exactly is going on in learners' brains as they view multimedia. Although this

may seem intuitive, neuroimaging is very rarely used in pedagogical research. To the author's

knowledge, EEG has been used in only two prior pedagogical investigations (Antonenko &

Niederhauser, 2010; Gerlio & Jaušovec, 1999). We decided to implement EEG methodology

into the present study not only because of the advantages it offers, but also to provide further

evidence that EEG is indeed a viable tool in educational research.

EEG is a non-invasive technique used to measure electrical activity in the brain via

electrodes placed on the scalp and can provide a continuous and objective measure of

cognitive activity (Picton, 2012). An advantage of EEG is its high temporal resolution, which

allows for the tracking of changes in brain activity on the millisecond scale (Picton, 2012).

Researchers can then determine fluctuations in cognitive activity at specific points of time,

9


such as when a stimulus is presented. This allows for a more accurate understanding of the

direct effect that some manipulation in multimedia design has on learning and cognition. In

this way, the use of EEG can compensate for the shortcomings of behavioural data.

While EEG is rarely used to measure cognitive load in a multimedia learning

environment, it is frequently used to measure cognitive load during the execution of cognitive

tasks not pertinent to pedagogy. One common approach is to look at brain wave activity in the

alpha band (8-13 Hz), as prior investigations have found that it varies with task difficulty, but

this variation is not predictable and consistent. Therefore, there are several hypotheses

regarding the functional significance of alpha and how exactly it is affected by task demands.

There are reports of alpha activity decreasing as cognitive load increases, which suggests

that alpha is inversely related to mental effort (Butler & Glass, 1976; Gerlio & Jaušovec,

1999). Confusingly, a substantial amount of studies have found just the opposite: that alpha

activity increases with task demands (Cooper et al. 2002; Galin et al., 1978; Handel et al.,

2011; Jensen et al., 2002; Osaka, 1984; Tuladhar et al., 2007). Researchers are divided in

their interpretation of these findings. For a number of years, the prevailing belief was that

alpha activity reflects cortical idling (see Pfurtscheller, Stancak, Neuper, 1996 for a review),

meaning that when a task is more difficult, subjects simply pay very little attention, and

cognitive activity is relatively diminished. In the past decade, however, support has increased

for the “alpha inhibition hypothesis” – the idea that alpha activity is an attentional suppression

mechanism. There is a growing body of evidence demonstrating increased alpha activity in

cortical regions that are involved in the processing of task-irrelevant stimuli, and that this is

correlated with better task performance (see Foxe & Snyder, 2011 for a review; Handel et al.,

10


2011; Jensen et al., 2002; Tuladhar et al., 2007). All in all, there is no consensus on what

exactly is reflected by alpha oscillations or how alpha is involved in learning.

The present study was primarily exploratory, with the overall goal of simply gaining a

better understanding of the cognitive processes involved in learning from multimedia – more

specifically, in an online lecture setting. We wanted to show that EEG methodology is useful

for evaluating the effects of multimedia design on cognitive processing and learning.

Furthermore, we sought to highlight how EEG can be used to measure brain activity in real-

life settings, and not just in laboratory-type cognitive tasks. Lastly, we strove to further

elucidate how alpha activity varies in response to changes in multimedia design and cognitive

load.

In order to minimize between-subject variability, we used a within-subjects design. We

ran EEG on subjects as they watched a simulated online lecture that included images and

narration. In order to increase the applicability of our findings, the lecture we presented to

subjects was an adaptation of a real online lecture used in McMaster's introductory

psychology course. In order to create two different types of lecture environments, half of the

lecture was manipulated to have greater cognitive load than the other. We did this by

scrambling the order in which the images appeared, thereby causing incongruence in audio

and visual stimuli, as irrelevant images increase the cognitive load of instructional media

(Mayer, Heiser, Lonn, 2001). We were primarily concerned with subjects' alpha power, which

is simply a measure of how much of the EEG signal contains waves oscillating within the

alpha band of frequency. We compared subjects' alpha power from each half of the lecture,

and hypothesized that it would be significantly greater in the half in which the images and

audio were incongruent. This is because attention towards the images needs to be

11


suppressed in order for subjects to gain a clear understanding of the aurally presented lecture

material. This hypothesis is in line with the view that alpha represents cortical inhibition (see

Foxe & Synder, 2011 for a review; Handel et al. 2011; Jensen et al., 2002; Tuladhar et al.,

2007).

Not only were we interested in seeing how audiovisual incongruence changes

cognitive activity, but we also wanted to determine how it affected subjects' learning.

Following the lecture, subjects completed a quiz measuring their knowledge of the lecture

material. The quiz contained questions that measured basic retention, while others required

application of the lecture's general principles. We hypothesized that subjects would perform

more poorly on application questions compared to recall questions overall, as they require a

deeper level of processing, which makes them naturally more challenging. In addition, we

predicted that scores would be lower on questions dealing with material from the incongruent

half of the lecture, regardless of question type, as the required inhibition of images should

leave fewer cognitive resources for attending to the audio track, which was the mode in which

the question material was presented.

12


Methods:

Participants:

Forty-six Psych 1XX3 students at McMaster University volunteered to be part of

the study in return for course credit. Subjects were all between 18 and 25 years of age,

in order to prevent age-related differences from confounding the EEG data.

Approximately twice as many females as males participated in the study. All subjects

completed the quiz (mean age = 19.09, SD = 1.49), but EEG was run on just eighteen

subjects, and only eight EEGs yielded analyzable data. Five EEGs were from female

subjects, and the other three were from males (mean age = 19.25, SD = 1.49). All

participants provided informed consent, and all procedures were reviewed by and

received clearance from the McMaster Research Ethics Board.

Stimuli & Procedure:

Lecture:

Subjects watched a nine-minute condensed adaptation of an actual

online lecture from Psych 1XX3, the second course in the introductory

psychology sequence at McMaster. The topic of the lecture was the

physiological and evolutionary bases of hunger and satiety. The lecture

consisted of 18 slides: each slide contained a single image with no text.

Participants viewed these images while simultaneously listening to a narration

that delivered the actual lecture content. For one half of the lecture, the images

appeared in the proper order and were congruent with the audio track, meaning

that they visually depicted a key aspect of what was being discussed at that

13


point of the lecture. However, for the other half of the lecture, the images were

scrambled. This meant that instead of the images matching the audio track in

terms of content, they corresponded to content that was discussed earlier or

later in the lecture. We counterbalanced for which half of the lecture had the

audiovisual incongruence. Half of the participants watched a version of the

lecture where the first half was incongruent, while the other half of the subjects

watched a version where the second half was incongruent. Prior to viewing,

subjects were informed that they would have to complete a quiz based on the

content of the lecture and instructed to pay close attention to the lecture.

EEG Recording & Analysis:

Participants were fitted with a 256-channel Hydrocel Geodesic Sensor

Net from EGI and seated in a soundproof booth 100 cm away from a computer

monitor. EEG was recorded by Net Station version 4.3.1 while subjects watched

the lecture. The signal was filtered with a 200 Hz low-pass filter and a 0.1 Hz

high-pass filter. The sampling rate was 500 Hz. Epochs were 14 seconds in

length – they begun 2 seconds prior and ended 12 seconds after the

presentation of a new image/slide. Welch’s Method was used to reduce noise

and estimate power within the alpha band. We first determined subjects' alpha

power at each of these slide changes. Then for each subject, we calculated an

average of their alpha power associated with the presentation of slides 1-9 and

10-18. This gave us the mean alpha power exhibited by each subject during

each half of the lecture. We then calculated the mean alpha power in the

14


congruent and incongruent halves of the lecture across all 8 subjects. Through

use of a topoplot, we determined that it was in the parietal region where

subjects showed the greatest alpha power in the incongruent half of the lecture

relative to the congruent half (see Figure 1). This disparity was most prominent

around E140, so we narrowed our scope of analysis to this single electrode, for

sake of simplicity.

Comprehension Quiz:

Immediately after viewing the lecture, subjects had to complete a

multiple-choice quiz in order to measure their comprehension of the lecture

material. There were 20 questions in total, with each question having four

possible options. In order to mimic a more realistic university-level evaluation

and to see how different types of cognitive skills were affected by the

incongruence, two different styles of questions were used. Half of the questions

measured basic retention of presented facts (recall), while the other half

required application of general principles to solve novel problems (application).

The quiz contained equal focus on material from each half of the lecture.

Examples of quiz questions are located in the Appendix. All participants were

briefed after completing of the quiz.

Statistical Analysis:

In order to compare the overall mean alpha power across all subjects during

each half of the lecture, we conducted a paired t-test. A mixed model repeated-

15


measures analysis of variance (ANOVA) was used to analyze comprehension quiz

scores. There were two within-subjects factors, each with two levels. One factor was

style of question, comparing application and recall questions. The other was origin of

question content; in other words, whether the question was derived from material

presented in the congruent or incongruent half of the lecture. In order to determine

whether subjects for whom the first half of the lecture was incongruent differed in quiz

performance from those who received the incongruent stimuli in the second half, we

included the between-subjects factor of condition.

16


Results:

Figure 2 shows the mean alpha power exhibited by subjects during each half of the

lecture at electrode E140. Overall, subjects showed greater alpha power when viewing the

incongruent half of the lecture compared to when they viewed the congruent half. This was

supported by a significant paired t-test contrasting mean alpha power within each lecture

halve with a moderate effect size [t(7) = 2.46, p = .043, d = .41].

Figure 3 illustrates performance on the comprehension quiz. Subjects performed

significantly better on recall-style questions than those that required application of knowledge

[F(1,44) = 75.97, p < .001]. Questions derived from the incongruent half of the lecture proved

more challenging for subjects than those derived from the congruent half, as the ANOVA

found a significant effect of question content origin [F(1,44) = 6.18, p = .017]. We did not find

an effect of condition [F(1,44) = .655, n.s.], meaning that there was no difference in quiz

performance between subjects for whom the first half of the lecture was incongruent and

those for whom the second half was incongruent. There was no significant interaction

between any of the factors [all Fs < 2.5].

17


Discussion:

As predicted, audiovisual incongruence in a multimedia presentation is detrimental to

learning. Subjects performed worse on quiz questions derived from the half of the lecture that

contained incongruent audio and visual stimuli. Collapsing across style of question, subjects

scored, on average, almost 7% lower on questions from the incongruent half, compared to

those taken from the congruent half. This difference in performance is significant as it

amounts to a difference of two grade points on most university grading scales. A possible

interpretation of these results is that subjects expended a large amount of cognitive resources

trying to resolve the incongruence between audio and visual stimuli, leaving them with an

insufficient amount of processing power to devote to the encoding of the lecture material.

It is also worth noting that the ANOVA found no effect of condition on quiz scores. This

means that subjects who received the incongruent stimuli first did not perform differently than

those for whom the second half of the lecture was incongruent. Taken together with effect of

question origin, this suggests that the initial audiovisual incongruence did not influence

subjects to merely ignore or not attend to the entirety of the presentation. It is likely that

subjects showed renewed interest when the images began to correspond with the audio track,

as reflected by higher quiz scores.

This is line with previous research demonstrating that inclusion of irrelevant stimuli in

instructional materials is detrimental to learning (Mayer, 2001; Mayer et al., 2001; Moreno &

Mayer, 2000). Learners seem especially sensitive to audiovisual incongruence, however. To

create incongruence, we merely scrambled the order of images, which means that although

the image did not directly correspond to the audio track at that time, it did coincide with what

was narrated a few minutes before or after. Thus, the images still related to the audio track in

18


overall theme – they all were related to some aspect of hunger, the topic of the lecture. This

shows that learning is harmed by even subtle disparities in audio and visual content in a

presentation, further stressing how important it is for instructors to select images very

carefully when preparing instructional materials.

We also found that subjects performed much worse on application questions than on

recall questions, regardless of which half of the lecture the questions were derived from. This

was expected because application is inherently more difficult than recall. Recall questions

measure basic retention and therefore, subjects needed only a surface knowledge of the

question content to correctly answer these types of questions. In contrast, application

questions evaluated subjects' ability to learn general principles and apply them to novel

contexts. In order to do well on questions of this style, subjects must have engaged in a

comparatively deeper level of processing and had a much more thorough understanding of

the material.

Not only did the quiz questions vary in their inherent difficulty, but so did the lecture

itself. Our key experimental manipulation was inducing cognitive load during one half of the

lecture by scrambling the order in which the images appeared. Our results indicate that this

manipulation was successful. Indirect support comes from subjects' lower quiz scores on

questions derived from this part of the lecture in combination with direct evidence from their

EEG data. Subjects showed significantly greater alpha power when the images were

incongruent with the accompanying narration. The disparity in alpha power between the two

halves of the lecture was most pronounced in the parietal region.

These results are in contrast to those from a prior multimedia study that used EEG

methodology (Gerlio & Jaušovec, 1999). Gerlio & Jaušovec (1999) found no difference in

19


alpha power at parietal electrodes when subjects were learning from a text-only presentation

or from multimedia, despite the difference in cognitive load between the two formats. The

disparity in results between the two studies is likely due to the use of different stimuli. The

parietal lobe is involved in the integration of audio and visual stimuli (Molholm et al., 2006),

and therefore, is more sensitive to the experimental manipulation used in our study.

Although the functional significance of alpha is highly debated in the literature, one

interpretation of our results lends support to the theory that alpha represents cortical

inhibition. When viewing educational multimedia, one must process both visual and auditory

stimuli and integrate them into a cohesive whole in order to gain a complete understanding of

the presented concepts. By presenting subjects with incongruent audiovisual stimuli, this not

only made it more difficult for subjects to integrate information from both sensory modalities,

but it made integration detrimental to learning. Because the images did not complement the

narration, the best learning would occur from ignoring the images and directing full attentional

capacity towards the audio track. In other words, during the incongruent half of the lecture,

the parietal lobe is not needed to carry out audiovisual integration. Thus, it is possible that the

increase of alpha power during this half of the lecture was a mechanism to inhibit audiovisual

integration in the parietal lobe, thereby providing subjects with more cognitive resources to

use towards auditory processing and making sense of the lecture content. This interpretation

is in line with previous findings that alpha activity increases in task-irrelevant regions (Handel

et al., 2011; Pfurtscheller & Klimesch, 1990). Following this logic, we would expect that alpha

activity would decrease in task-relevant regions, and therefore, subjects would show much

less alpha power in temporal regions than in parietal regions during the incongruent half of

the lecture. However, this was beyond our scope of analysis. Furthermore, we would expect

20


higher alpha power during the incongruent half of the lecture would lead to better

performance on questions pertaining to that portion of the lecture. However, due to our small

sample size, we could not compute a correlation between quiz scores and alpha power.

Therefore, we cannot state for certain that our results support the alpha inhibition hypothesis.

We cannot rule out the possibility that higher alpha power during the incongruent half of the

lecture actually reflects cortical idling. If subjects are not integrating what they see with what

they hear, the parietal lobe could merely be in a state of relative cognitive inactivity.

The overarching goal of this study was to explore a real-life application of EEG by

exploring how it can inform pedagogical research. Our results demonstrate that EEG can

indeed provide insight into the cognitive processes involved in multimedia learning.

Specifically, EEG shows promise as a reliable measure of cognitive load during authentic

learning situations, as we observed significantly higher alpha power in the incongruent half of

the lecture. As cognitive load is an important consideration in multimedia design, this

suggests that EEG may be a useful tool when interpreting the effectiveness of different

formats and designs of multimedia presentations. By complementing behavioural measures

with EEG data, we can obtain a greater breadth of understanding in regards to multimedia

learning.

Although our study highlights the value of using EEG in educational research, our

results should be interpreted with caution. Despite running EEG on many more subjects, only

eight EEGs yielded analyzable data, which limits the generalizability of our findings. In order

to get a more reliable measure of how alpha power changes in response to audiovisual

incongruence, this study should be repeated with a greater number of subjects. In addition,

we were only able to analyze alpha power at a single electrode in the parietal area. In order to

21


more accurately assess how parietal alpha power varies with cognitive load, follow-up studies

should calculate the mean alpha power across all electrodes in the parietal region. This

should also be done for all other brain regions, so that we can gain insight into the topography

of alpha activity, as this would be useful in elucidating alpha's functional significance in

multimedia learning, and its role in general cognitive performance.

Despite these limitations, our study achieved its overall purpose of exemplifying that

pedagogical research can benefit from the implementation of methodology from cognitive

neuroscience. EEG and other types of neuroimaging techniques can provide insight into the

neural correlates of multimedia learning and enable researchers to investigate a wider variety

of research questions. However, further work is needed to determine how exactly data

obtained using neuroscience techniques can be used to inform multimedia design and

instruction. While this study provides a glimpse into the cognitive processes that underlie

learning, there is much left to be discovered. By combining findings and perspectives from

cognitive neuroscience and pedagogy, we can gain a more thorough understanding of this

complex topic. This study provides great incentive of a merger between these two fields of

psychology and it will be exciting to see how they flourish and interact with each other in

years to come.

22


Figures:

Figure 1.Topoplot comparing differences in alpha power. Regions in red represent the

brain areas in which alpha power was much greater in the incongruent half of the lecture than

in the congruent half; the circled area represents the area where this difference was greatest.

This is the area corresponding to electrode E140. Regions in blue represent regions in which

the alpha power was much lower in the incongruent half of the lecture compared to the

congruent half, but this was outside our scope of analysis.

23


Figure 2. Mean alpha power exhibited during each half of the lecture (± SE).

24


Figure 3. Mean comprehension quiz scores by question type and origin (± SE).

25


References:

Apperson, J. M., Laws, E. L., & Scepansky, J. A. (2008). An assessment of student

preferences for PowerPoint presentation structure in undergraduate courses.

Computers & Education, 50(1), 148-153.

Baddeley, A. (1992). Working memory. Science, 255(5044), 556-559.

Bartsch, R.A., Cobern, K.M. (2003). Effectiveness of PowerPoint presentations in lectures.

Computers & Education, 41, 77–86.

Beeland, W. D. (2002). Student engagement, visual learning and technology: Can

interactive whiteboards help. In Annual Conference of the Association of Information

Technology for Teaching Education.

Bruenken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in

multimedia learning. Educational Psychologist, 38(1), 53-61.

Butler, S. R., & Glass, A. (1976). EEG correlates of cerebral dominance. Advances in

Psychobiology, 3, 219.

Cooper, N. R., Croft, R. J., Dominey, S. J., Burgess, A. P., & Gruzelier, J. H. (2003). Paradox

lost? Exploring the role of alpha oscillations during externally vs. internally directed

attention and the implications for idling and inhibition hypotheses. International Journal

of Psychophysiology, 47(1), 65-74.

Cutrim Schmid, E. (2008). Potential pedagogical benefits and drawbacks of multimedia use in

the English language classroom equipped with interactive whiteboard technology.

Computers & Education, 51(4), 1553-1568.

26


Foxe, J. J., & Snyder, A. C. (2011). The role of alpha-band brain oscillations as a sensory

suppression mechanism during selective attention. Frontiers in Psychology, 2.

Galin, D., Johnstone, J., & Herron, J. (1978). Effects of task difficulty on EEG measures of

cerebral engagement. Neuropsychologia, 16(4), 461-472.

Gerlio, I., & Jaušovec, N. (1999). Multimedia: Differences in cognitive processes observed

with EEG. Educational Technology Research and Development, 47(3), 5-14.

Jacques, R. (1995). Engagement as a Design Concept for Multimedia. Canadian Journal of

Educational Communication, 24(1), 49-59.

Jensen, O., Gelfand, J., Kounios, J., & Lisman, J. E. (2002). Oscillations in the alpha band (9–

12 Hz) increase with memory load during retention in a short-term memory task.

Cerebral Cortex, 12(8), 877-882.

Kornell, N., Bjork, R.A. (2008). Learning concepts and categories: is spacing the “enemy

of induction”? Psychological Science, 19(6), 585–592.

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.

Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning:

When presenting more material results in less understanding. Journal of Educational

Psychology, 93(1), 187-198.

Molholm, S., Sehatpour, P., Mehta, A. D., Shpaner, M., Gomez-Ramirez, M., Ortigue, S., ... &

Foxe, J. J. (2006). Audio-visual multisensory integration in superior parietal lobule

revealed by human intracranial recordings. Journal of neurophysiology, 96(2), 721-729.

Moreno, R., & Mayer, R. E. (2000). A coherence effect in multimedia learning: The case for

minimizing irrelevant sounds in the design of multimedia instructional messages.

Journal of Educational Psychology, 92(1), 117-125.

27


Najjar, L. J. (1996). Multimedia information and learning. In Journal of Educational Multimedia

and Hypermedia.

Osaka, M. (1984). Peak alpha frequency of EEG during a mental task: Task difficulty and

hemispheric differences. Psychophysiology, 21(1), 101-105.

Pfurtscheller, G., & Klimesch, W. (1990). Topographical display and interpretation of event-

related desynchronization during a visual-verbal task. Brain Topography, 3(1), 85-93.

Pfurtscheller, G., Stancak Jr, A., & Neuper, C. (1996). Event-related synchronization (ERS) in

the alpha band—an electrophysiological correlate of cortical idling: a review.

International Journal of Psychophysiology, 24(1), 39-46.

Picton, T. (2012). What is an EEG? Retrieved from http://research.baycrest.org/eeg.

Shuell, T. J., & Farber, S. L. (2001). Students' perceptions of technology use in college

courses. Journal of Educational Computing Research, 24(2), 119-138.

Tuladhar, A. M., Huurne, N. T., Schoffelen, J. M., Maris, E., Oostenveld, R., & Jensen, O.

(2007). Parieto-occipital sources account for the increase in alpha activity with working

memory load. Human brain mapping, 28(8), 785-792.

28


Appendix:

Sample application-style question:

20. Alex, Sam, and Amanda all eat the same amount of food for dinner. Alex eats a triple cheeseburger with fries. Sam eats a large turkey sandwich with a plate of steamed vegetables. Amanda eats a large plate of chicken thighs and wings. Who will feel full the longest?

a. Alexb. Amandac. Samd. They will all feel full for the same amount of time

Sample recall-style question:

10. Which nutrient signals the need to replenish one’s food intake?

a. glucoseb. fructosec. adipose tissued. glycogen

audiovisual incongruence in multimedia: an exploratory eeg study

Documents