a formative evaluation of the training effectiveness of a ...€¦ · web viewgoal orientation...

Navigation Maps and Problem Solving: revised 5/8/05

The Effect of Navigation Maps on Problem Solving Tasks

Instantiated in a Computer-Based Video Game

Dissertation Proposal Presented to the

Faculty of the Graduate School

University of Southern California

Richard Wainess

University of Southern California

to

Dr. Harold O’Neil (Chair)

Dr. Richard Clark

Dr. Edward Kazlauskas

Dr. Janice Schafrik

Dr. Yanis Yortsos (Outside Member)

14009 Barner Ave., Sylmar, CA 91342

(818) 364-9419

[email protected]

In Partial Fulfillment for Ph.D. in Educational Psychology and Technology

April 21, 2004

1


Table of Contents

ABSTRACT...............................................................................................................................6

CHAPTER I: INTRODUCTION...............................................................................................7

Background of the Problem.......................................................................................................7

Statement of the Problem...........................................................................................................9

Purpose of the Study..................................................................................................................9

Significance of the Study.........................................................................................................10

Research Hypotheses and Questions.......................................................................................11

Overview of the Methodology.................................................................................................12

Assumptions.............................................................................................................................12

Limitations ..........................................................................................................................13

Delimitations............................................................................................................................13

Organization of the Report.......................................................................................................14

CHAPTER II: LITERATURE REVIEW................................................................................15

Cognitive Load Theory............................................................................................................16

Working Memory.........................................................................................................17

Long Term Memory.....................................................................................................17

Schema Development...................................................................................................18

Mental Models..................................................................................................19

Elaboration and Reflection...............................................................................20

Meaningful Learning....................................................................................................20

Metacognition...................................................................................................21

Cognitive Strategies..............................................................................21

2


Mental Effort and Persistence...........................................................................22

Goals.................................................................................................................22

Goal Setting Theory..............................................................................22

Goal Orientation Theory.......................................................................23

Self-Efficacy.....................................................................................................23

Self-Efficacy Theory.............................................................................24

Expectancy-Value Theory....................................................................25

Task Value............................................................................................25

Problem Solving...............................................................................................26

O’Neil’s Problem Solving Model.........................................................27

Types of Cognitive Loads............................................................................................28

Learner Control............................................................................................................31

Summary of Cognitive Load........................................................................................32

Games and Simulations............................................................................................................34

Games...........................................................................................................................35

Simulations...................................................................................................................36

Simulation-Games........................................................................................................37

Motivational Aspects of Games...................................................................................38

Fantasy..............................................................................................................39

Control and Manipulation.................................................................................40

Challenge and Complexity...............................................................................41

Curiosity...........................................................................................................41

Competition......................................................................................................42

3


Feedback...........................................................................................................43

Fun....................................................................................................................43

Learning and Other Outcomes for Games....................................................................45

Positive Outcomes from Games and Simulations............................................45

Negative or Null Outcomes from Games and Simulations..............................47

Relationship of Instructional Design to Effective Games and Simulations.....47

Reflection and Debriefing................................................................................49

Summary of Games and Simulations..........................................................................49

Assessment of Problem Solving..............................................................................................52

Measurement of Content Understanding......................................................................52

Measurement of Problem Solving Strategies............................................................56

Measurement of Self-Regulation..............................................................................57

Summary of Problem Solving Assessment..................................................................58

Scaffolding ..........................................................................................................................59

Graphical Scaffolding..................................................................................................60

Navigation Maps...............................................................................................61

Contiguity Effect..............................................................................................64

Split-Attention Effect.......................................................................................65

Summary of Scaffolding..............................................................................................65

Summary of the Literature Review..........................................................................................67

CHAPTER III: METHODOLOGY........................................................................................72

Research Design.................................................................................................................72

Research Questions and Hypotheses.......................................................................................72

4


Sample ..........................................................................................................................73

Instruments...............................................................................................................................73

Demographic, Game Preference, and Task Completion Questionnaire.......................74

Self-Regulation Questionnaire.....................................................................................74

SafeCracker..................................................................................................................75

Navigation Map............................................................................................................77

Knowledge Map...........................................................................................................78

Content Understanding Measure..................................................................................78

Domain-Specific Problem-Solving Strategies Measure...............................................80

Procedure for the Study...........................................................................................................81

Timing Chart................................................................................................................82

Data Analysis...........................................................................................................................83

REFERENCES........................................................................................................................85

APPENDICES

Appendix A: Self-Regulation Questionnaire.............................................................104

Appendix B: SafeCracker® Expert Map...................................................................106

Appendix C: Knowledge Map Specifications............................................................107

5


ABSTRACT

Cognitive load theory defines a limited capacity working memory with associated auditory and

visual/spatial channels. Navigation in computer-based hypermedia and video game environments

is believed to place a heavy cognitive load on working memory. Current 3-dimensional

computer-based video games (3-D, computer-based video games) often include complex

occluded environments (conditions where vision is blocked by objects in the environment, such

as internal walls, trees, hills, or buildings) preventing players from plotting a direct visual course

from the start to finish locations. Navigation maps may provide the support needed to effectively

navigate in these environments. Navigation maps are a type of graphical scaffolding, and

scaffolding, including graphical scaffolding, helps learners by reducing the amount of cognitive

load placed on working memory. Navigation maps have been shown to be effective in 3-D,

occluded, video game environments requiring complex navigation and simple problem solving

tasks. Navigation maps have also been shown to be effective in 2-dimensional environments

involving complex problem solving tasks. This study will extend the research by combining

these two topics—navigation maps for navigation in 3-D, occluded, computer-based video

games and navigation maps in 2-dimensional environments with complex problem solving tasks

—by examining the effect of a navigation map on a 3-D, occluded, computer-based video game

problem solving task. In addition, the effect of a navigation map on motivation will be examined.

6


CHAPTER I: INTRODUCTION

With the current power of computers and the current state-of-the-art of video games, it is

likely that future versions of educational video games will include immersive environments in

the form of 3-D, computer-based video games requiring navigation through occluded paths in

order to perform complex problem solving tasks. According to Cutmore, Hine, Maberly,

Langford, and Hawgood (2000), occlusion refers to conditions where vision is blocked by

objects in the environment, such as internal walls or large environmental features like trees, hills,

or buildings. Under these conditions, one cannot simply plot a direct visual course from the start

to finish locations (Cutmore et al., 2000). This study examines the use of navigation maps to

support navigation through a 3-D, occluded, computer-based video game involving a complex

problem-solving task.

Chapter one begins with an examination of the background of the problem. Next the

purpose of the study is discussed, followed by why the study is significant—how it will inform

the literature—and the hypotheses that will be addressed. The next sections in chapter one

include an overview of the methodology that will be utilized, assumptions that inform this topic,

study limitations and delimitations, and a brief explanation of the organization of this proposal.

Background of the Problem

Educators and trainers began to take notice of the power and potential of computer games

for education and training back in the 1970s and 1980s (Donchin, 1989; Malone, 1981; Malone

& Lepper, 1987; Ramsberger, Hopwood, Hargan, & Underfull, 1983; Ruben, 1999; Thomas &

Macredie, 1994). Computer games were hypothesized to be potentially useful for instructional

purposes and were also hypothesized to provide multiple benefits: (a) complex and diverse

approaches to learning processes and outcomes; (b) interactivity; (c) ability to address cognitive

7


as well as affective learning issues; and perhaps most importantly, (d) motivation for learning

(O’Neil, Baker, & Fisher, ,2002).

Research into the effectiveness of games and simulations as educational media has been

met with mixed reviews (de Jong & van Joolingen, 1998; Garris, Ahlers, & Driskell, 2002). It

has been suggested that the lack of consensus can be attributed to weaknesses in instructional

strategies embedded in the media and issues related to cognitive load (Chalmers, 2003; Cutmore,

Hine, Maberly, Langford, & Hawgood, 2000; Lee, 1999; Thiagarajan, 1998; Wolfe, 1997).

Cognitive load theory suggests that learning involves the development of schemas (Atkinson,

Derry, Renkl, & Wortham, 2000), a process constrained by a limited working memory with

separate channels for auditory and visual/spatial stimuli (Brunken, Plass, & Leutner, 2003).

Further, cognitive load theory describes an unlimited capacity, long-term memory that can store

vast numbers of schemas (Mousavi, Low, & Sweller, 1995).

The inclusion of scaffolding, which provides support during schema development by

reducing the load in working memory, is a form of instructional design; more specifically, it is an

instructional strategy (Allen, 1997; Clark, 2001). For example, graphical scaffolding, which

involves the use of imagery-based aids, has been shown to be an effective support for

graphically-based learning environments, including video games (Benbasat & Todd, 1993;

Farrell & Moore, 2000-2001; Mayer, Mautone, & Prothero, 2002). Navigation maps, a particular

form of graphical scaffolding, have been shown to be an effective scaffold for navigation of a 3-

dimensional (3-D) virtual environment (Cutmore et al., 2000). Navigation maps have also been

shown to be an effective support for navigating and problem-solving in a 2-dimension (2-D)

hypermedia environment (Baylor, 2001; Chou, Lin, & Sun, 2000), which is made up of nodes of

information and links between the various nodes (Bowdish, & Lawless, 1997). What has not

8


been examined, and is the purpose of this study, is the effect of navigation maps, utilized for

navigation in a 3-D, occluded, computer-based video game, on outcomes in a complex problem-

solving task.

Statement of the Problem

A major instructional issue in learning by doing within simulated environments concerns

the proper type of guidance, that is, how best to create cognitive apprenticeship (Mayer,

Mautone, & Prothero, 2002). A virtual environment creates a number of issues with regards to

learning. Problem-solving within a virtual environment involves not only the cognitive load

associated with the to-be-learned material (referred to as intrinsic cognitive load: Paas,

Tuovinen, Tabbers, Van Gerven, 2003), it also includes cognitive load related to the visual

nature of the environment (referred to as extraneous cognitive load: Brunken, Plass, & Leutner;

Harp & Mayer, 1998), as well as navigating within the environment—either germane cognitive

load or extraneous cognitive load, depending on the relationship of the navigation to the learning

task (Renkl, & Atkinson, 2003). An important goal of instructional design within these

immersive environments involves determining methods for reducing the extraneous cognitive

load and/or germane cognitive load, thereby providing more working memory capacity for

intrinsic cognitive load (Brunken et al., 2003). This study will examine the reduction of cognitive

load, by providing graphical scaffolding in the form of a navigation map, to determine if this can

result in better performance outcomes as reflected in retention and transfer (Paas et al., 2003) in a

game environment.

Purpose of the Study

The purpose of this study is to examine the effect of a navigation map on a complex

problem solving task in a 3-D, occluded, computer-based video game. The environment for this

9


study is the interior of a mansion as instantiated in the game SafeCracker® (Daydream

Interactive, 1995). The navigation map is a printed version of the floor plan of the first floor,

with relevant room information, such as the name of the room. The problem solving task

involves navigating through the environment to locate specific rooms, to find and acquire items

and information necessary to open safes located within the prescribed rooms, and ultimately, to

open the safes. With one group playing the game while using the navigation map and the other

group playing the game without aid of a navigation map, this study will exam differences in

problem solving outcomes informed by the problem solving model defined by O’Neil (1999).

Significance of the Study

Research has examined the use of navigation maps, a particular form of graphical

scaffolding, as navigation support for complex problem-solving tasks within a hypermedia

environment, with the navigation map providing an overview of the 2-dimension structure which

had been segmented into nodes (Chou, Lin, & Sun, 2000). Research has also examined the use of

navigation maps as a navigational tool in 3-D virtual environments, but has only examined the

effect of the navigation map on navigation (Cutmore, Hine, Maberly, Langford, & Hawgood,

2000) or during a complex navigation task that involved a very basic problem solving task;

finding a key along the path in order to open a door at the end of the path (Galimberti, Ignazi,

Vercesi, &Riva, 2001). Research has not combined these two research topics; it has not

examined the use of navigation maps in relationship to a complex problem-solving task that

involved navigation within a complex 3-D virtual environment.

While a number of studies on hypermedia environments have examined the issue of site

maps to aid in navigation of the various nodes for problem solving tasks (e.g., Chou & Lin,

1998), no study has looked at the effect of the use of two-dimensional topological maps (a floor

10


plan) for navigation within a 3-dimensional video game environment in relationship to complex

problem solving task. It is argued here that the role of the two navigation map types (site map

and topological floor plan) serve the same purpose in terms of cognitive load. However, it is also

argued here that the spatial aspect of the two learning environments differ significantly, placing a

larger load on the visual/spatial channel of working memory with a 3-D video game environment

as compared to a 2-D hypermedia environment, thereby leaving less working memory capacity

in the 3-D video game for visual stimuli; the navigation map.

As immersive 3-D video games are becoming more common as commercial video games,

it is likely they will also become more common as educational media. Therefore, the role of

navigation maps to reduce the load induced by navigation and, therefore, reduce burdens on

working memory, is an important issue for enhancing the effectiveness of games as educational

environments.

Research Question and Hypotheses

Research Question 1: Will the problem solving performance of participants who use a

navigation map in a 3-D, occluded, computer-based video game (i.e., SafeCracker®) be better

than the problem solving performance of those who do not use the map (the control group)?

Hypothesis 1: Navigation maps will produce a significant increase in content

understanding compared to the control group.

Hypothesis 2: Navigation maps will produce a significant increase in problem solving

strategy retention compared to the control group.


strategy transfer compared to the control group.

11


Hypothesis 4: There will be no significant difference in self-regulation between the

navigation map group and the control group. However, it is expected that higher levels of self-

regulation will be associated with better performance.

Research Question 2: Will the continued motivation of participants who use a

navigation map in a 3-D, occluded, computer-based video game (i.e., SafeCracker®) be greater

than the continued motivation of those who do not use the map (the control group)?

Hypothesis 5: Navigation maps will produce a significantly greater amount of optional

continued game play compared to the control group.

Overview of the Methodology

The design of this study is an experimental with pre-, intermediate-, and post-tests for one

treatment group and one control group. Subjects will be randomly assigned to either the

treatment or the control group. Group sessions will involve only one group type: either all

treatment participants or all control participants. The experimental design involves

administration of pretest instruments, the treatment, administration of intermediate test

instruments, the treatment, and administration of the posttest instruments. At the end of the

session, participants will be debriefed and will be then allowed to continue playing on their own

for up to 30 additional minutes (to examine continued motivation).

Assumptions

This research assumes the acceptance of the cognitive load theory, which describes a

limited working memory with separate auditory and visual-spatial channels, an unlimited long-

term memory, and the existence of schema. Also assumed is the influence of scaffolding on

reducing cognitive load, by assisting in the development of schema and in distributing cognitive

load.

12


Limitations

There are a number of potential weaknesses to the current study. First, is the nature of the

main instrument; the game SafeCracker®. The game was created in 1995 and, by current

standards for game graphics and dynamics, is not a particularly engaging game. While it is

assumed this issue will not affect problem-solving assessments, it is assumed it will likely

influence motivational outcomes—specifically, continued motivation as assessed by the desire to

continue playing the game after the experimental procedure is completed.

Another weakness is the introduction of both the contiguity effect (Mayer, Moreno,

Boire, & Vagge, 1999; Mayer & Sims, 1994; Moreno & Mayer, 1999), in the form of spatial

contiguity, and the split-attention effect (Mayer & Moreno, 1998; Yeung, Jin, & Sweller, 1997)

into the study; for the treatment group. Because the game is visual and will seen on a computer

screen, and the navigation map (the floor plan) given to the treatment group, also visual but in

printed format, will be spatially separate from the screen, cognitive load issues as described by

both the contiguity effect and the split-attention effect will be imposed. This study will not

examine the impact of this added load, and how it might influence learning outcomes, possibly

offsetting the benefits of the graphical scaffolding (the navigation map).

Delimitations

All participants will be undergraduate students of a tier-1 university; one of the top one

hundred universities in the United States. This suggests that the sample will not easily generalize

to a larger population. Second, since participants are all volunteers, it is likely they enjoy playing

video games. Therefore, they represent only the video game playing population, and study results

cannot be generalized to those who are less inclined to play or to volunteer to play video games.

A third delimitation is the time period of the study; summer. During summer, a large portion of

13


the student population is not on campus. Therefore, those who respond to the flyer, or other

marketing means for soliciting study participants, will not be a true representation of the full

student population. Combined, these delimitations suggest that the sample enlisted for this study

will be bar generalization beyond a narrow population.

Organization of the Report

Chapter one provides an overview of the study with a brief introduction and background

for the topic, the problem being addressed, the significance of the study, the hypotheses that will

be tested, an overview of the methodology of the experiments, and assumptions, limitations, and

delimitations related to the study. Chapter two is the literature review of the domains that inform

the current research: cognitive load theory, games and simulations, assessment of problem-

solving, and scaffolding. Chapter three describes the study’s methodology, with discussions of

the population, the sample, the study, the instruments, the procedures, and the proposed data

analysis methods.

14


CHAPTER II: LITERATURE REVIEW

The literature review includes information on four areas relevant to the research topic:

cognitive load theory, games and simulations, assessment of problem solving, and scaffolding.

The cognitive load section is comprised of an introduction to cognitive load, followed by

discussions of working and long-term memory, schema development, and mental models and the

role of reflection and elaboration. Next, under cognitive load theory, is a discussion of

meaningful learning, including the role of metacognition and cognitive strategies, mental effort

and persistence, goals and related theories, self-efficacy and several related theories and topics,

and problem solving, with a discussion of O’Neil’s Problem Solving model (O’Neil, 1999).

Types of cognitive load are then discussed—specifically, intrinsic cognitive load, germane

cognitive load, and extraneous cognitive load—as well as learner control as informed by

cognitive load theory. The cognitive load theory section ends with a summary of the section.

Following cognitive load theory is a discussion of games and simulations, beginning with

defining games, simulations, and simulation-games. Next the motivational aspects of games is

introduced with a discussion of the major characteristics of motivation: fantasy, control and

manipulation, challenge and complexity, curiosity, competition, feedback, and fun. The final

section under games and simulations is learning and other outcomes, which ends with a short

discussion of the role of reflection and debriefing. Last is a summary of games and simulations.

The third section of chapter two is the assessment of problem solving focused on the three

constructs established in O’Neil’s Problem Solving model (O’Neil, 1999): measurement of

content understanding, measurement of problem solving strategies, and measurement of self-

regulation. The section ends in with a summary of problem solving assessment.

15


The fourth and final section, scaffolding, begins with a general discussion of scaffolding,

followed by a review of the literature on a type of scaffolding relevant to this study, graphical

scaffolding. Within graphical scaffolding, navigation maps are examined, along with the

relationship of the contiguity effect and the split attention effect to inclusion of a navigation map.

The section ends in with a summary of scaffolding. The chapter ends with a summary of chapter

two.

Cognitive Load Theory

Cognitive load theory, which began in the 1980s and underwent substantial development

and expansion in the 1990s (Paas, Renkl, & Sweller, 2003), is concerned with the development

of instructional methods aligned with the learners’ limited cognitive processing capacity, to

stimulate their ability to apply acquired knowledge and skills to new situations (i.e., transfer).

Brunken, Plass, and Leutner (2003) argued that cognitive load theory is based on several

assumptions regarding human cognitive architecture: the assumption of a virtually unlimited

capacity of long-term memory, schema theory of mental representations of knowledge, and

limited-processing capacity assumptions of working memory (Brunken et al., 2003). Cognition is

the intellectual processes through which information is obtained, represented mentally,

transformed, stored, retrieved, and used. cognitive load theory is based on the idea that a

cognitive architecture exists consisting of a limited working memory, with partly independent

processing units for visual-spatial and auditory-verbal information (Mayer & Moreno, 2003), and

these structures interact with a comparatively unlimited long-term memory (Mousavi, Low, &

Sweller, 1995).

Cognitive load is the total amount of mental activity imposed on working memory at an

instance in time (Chalmers, 2003; Cooper, 1998; Sweller and Chandler, 1994, Yeung, 1999).

16


Researchers have proposed that working memory limitations can have an adverse effect on

learning (Sweller and Chandler, 1994, Yeung, 1999). According to Paas, Tuovinen, Tabbers, &

Van Gerven, (2003), cognitive load can be defined as a multidimensional construct representing

the load that performing a particular task imposes on the learner’s cognitive system. The

construct has a causal dimension reflecting the interaction between task and learner

characteristics, and an assessment dimension reflecting the measurable concepts of mental load,

mental effort, and performance (Paas et al., 2003). Cognitive load is a theoretical construct,

describing the internal processes of information processing that cannot be observed directly

(Brunken et al., 2003).

Working Memory

Working memory refers to the limited capacity for holding information in mind for several

seconds in the context of cognitive activity (Gevins et al., 1998). According to Brunken et al.

(2003), the Baddeley (1986) model of working memory assumes the existence of a central

executive that coordinates two slave systems, a visuospatial sketchpad for visuospatial

information such as written text or pictures, and a phonological loop for phonological

information such as spoken text or music (Baddeley, 1986, Baddeley & Logie, 1999). Both slave

systems are limited in capacity and independent from one another so that the processing

capacities of one system cannot compensate for lack of capacity in the other (Brunken et al.,

2003).

Long-Term Memory

According to Paas et al. (2003), working memory, in which all conscious cognitive

processing occurs, can handle only a very limited number of novel interacting elements; possibly

no more than two or three. In contrast, long-term memory has an unlimited, permanent capacity

17


(Tennyson & Breuer, 2002) and can contain vast numbers of schemas—cognitive constructs that

incorporate multiple elements of information into a single element with a specific function (Paas

et al., 2003). Noyes and Garland (2003) contended that information that is not held in working

memory will need to be retained by the long-term memory system. Storing more knowledge in

long-term memory reduces the load on working memory, which results in a greater capacity

being made available for active processing.

According to cognitive load theory, multiple elements of information can be chunked as

single elements in cognitive schema (Chalmers, 2003), and through repeated use can become

automated. Automated information, developed over hundreds of hours of practice (Clark, 1999),

can be processed without conscious effort, bypass working memory during mental processing,

thereby circumventing the limitations of working memory (Clark 1999; Mousavi et al., 1995).

Consequently, the primay goals of instruction are the construction (chunking) and automation of

schemas (Paas et al., 2003).

Schema Development

Schema is defined as a cognitive construct that permits people to treat multiple sub-

elements of information as a single element, categorized according to the manner in which it will

be used (Kalyuga, Chandler, & Sweller, 1998). Schemas are generally thought of as ways of

viewing the world and, in a more specific sense, ways of incorporating instruction into our

cognition. Schema acquisition is a primary learning mechanism. Piaget proposed that learning is

the result of forming new schemas and building upon previous schemas (as cited in Chalmers,

2003). Schemas have the functions of storing information in long-term memory and of reducing

working memory load by permitting people to treat multiple elements of information as a single

element (Kalyuga, et al., 1998; Mousavi et al., 1995).

18


With schema use, a single element in working memory might consist of a large number of

lower level, interacting elements which, if processed individually, might have exceeded the

capacity of working memory (Paas et al., 2003). If a schema can be brought into working

memory in automated form, it will make limited demands on working memory resources,

leaving more resources available to search for a possible solution problem (Kalyuga et al., 1998).

Controlled use of schemas requires conscious effort, and therefore, working memory resources.

However, after being sufficiently practiced, schemas can operate under automatic, rather than

controlled, processing. Automatic processing of schemas requires minimal working memory

resources and allows for problem solving to proceed with minimal effort (Kalyuga, Ayers,

Chandler, & Sweller, 2003; Kalyuga et al., 1998; Paas et al., 2003).

Mental Models

Mental models explain human understanding of external reality, translating reality into

internal representations and utilizing them in problem solving (Park & Gittelman, 1995).

According to Allen (1997), mental models are usually considered the way in which people model

processes. This emphasis on process distinguishes mental models from other types of cognitive

organizers such as schemas. A mental model synthesizes several steps of a process and organizes

them as a unit. A mental model does not have to represent all of the steps which compose the

actual process (Allen, 1997). Mental models may be incomplete and may even be internally

inconsistent. Models of mental models are termed conceptual models. Conceptual models

include: metaphor; surrogates; mapping, task-action grammars, and plans. Mental model

formation depends heavily on the conceptualizations that individuals bring to a task (Park &

Gittelman, 1995).

19


Elaboration and Reflection

Elaboration and reflection are processes involved to the development of schemas and

mental models. Elaborations are used to develop schemas whereby nonarbitrary relations are

established between new information elements and the learner’s prior knowledge (van

Merrienboer, Kirshner, & Kester, 2003). Elaboration consists of the creation of a semantic event

that includes the to-be-learned items in an interaction (Kees & Davies, 1990). With reflection,

learners are encouraged to consider their problem-solving process and to try to identify ways of

improving it (Atkinson, Renkl, & Merrill, 2003). Reflection is reasoned and conceptual, allowing

the thinker to consider various alternatives (Howland, Laffey, & Espinosa, 1997). According to

Chi (2000) the self-explanation effect (aka reflection or elaboration) is a dual process that

involves generating inferences and repairing the learner’s own mental model.

Meaningful Learning

Meaningful learning is defined as deep understanding of the material, which includes

attending to important aspects of the presented material, mentally organizing it into a coherent

cognitive structure, and integrating it with relevant existing knowledge (Mayer & Moreno,

2003). Meaningful learning is reflected in the ability to apply what was taught to new situations;

problem solving transfer. Meaningful learning results in an understanding of the basic concepts

of the new material through its integration with existing knowledge (Davis, & Wiedenbeck,

2001).

According to assimilation theory, there are two kinds of learning: rote learning and

meaningful learning. Rote learning occurs through repetition and memorization. It can lead to

successful performance in situations identical or very similar to those in which a skill was

initially learned. However, skills gained through rote learning are not easily extensible to other

20


situations, because they are not based on deep understanding of the material learned. Meaningful

learning, on the other hand, equips the learner for problem solving and extension of learned

concepts to situations different from the context in which the skill was initially learned (Davis, &

Wiedenbeck, 2001; Mayer, 1981).Meaningful learning takes place when the learner draws

connections between the new material to be learned and related knowledge already in long-term

memory, known as the assimilative context (Ausubel, 1963; Davis, & Wiedenbeck, 2001).

Meaningful learning requires mental effort. According to Clark (2003b), “mindful” mental effort

requires instructional messages (feedback) that point out the novel elements of the to-be-learned

material and must emphasize the need to work hard. Instructional messages must present be

concrete and challenging, yet achievable, learning and performance goals. The following

sections address the majority of these topics.

Metacognition

Metacognition, or the management of cognitive processes, involves goal-setting, strategy

selection, attention, and goal checking (Jones, Farquhar, & Surry, 1995). According to Harp and

Mayer (1998), many cognitive models include the executive processes of selecting, organizing,

and integrating. Selecting involves paying attention to the relevant pieces of information.

Organizing involves building internal connections among the selected pieces of information,

such as causal chains. Integrating involves building external connections between the incoming

information and prior knowledge existing in the learner’s long-term memory (Harp & Mayer,

1998).

Cognitive strategies. Cognitive strategies include rehearsal strategies, elaboration

strategies, organization strategies, affective strategies, and comprehension monitoring strategies.

These strategies are cognitive events that describe the way in which we process information

21


(Jones et al., 1995). Metacognition is a type of cognitive strategy that has executive control over

other cognitive strategies. Prior experience in solving similar tasks and using various strategies

will affect the selection of a cognitive strategy (Jones et al., 1995).

Mental Effort and Persistence

Mental effort is the aspect of cognitive load that refers to the cognitive capacity that is

actually allocated to accommodate the demands imposed by a task; thus, it can be considered to

reflect the actual cognitive load. Mental effort, relevant to the task and material, appears to be the

feature that distinguishes between mindless or shallow processing on the one hand, and mindful

or deep processing, on the other. Little effort is expended when processing is carried out

automatically or mindlessly (Salomon, 1983). Motivation generates the mental effort that drives

us to apply our knowledge and skills. According to Clark (2003d), “Without motivation, even the

most capable person will not work hard” (p. 21). However, mental effort investment and

motivation should not be equated. Motivation is the driving force, but for learning to actually

take place, some specific relevant mental activity needs to be activated. This activity is assumed

to be the employment of nonautomatic effortful elaborations (Salomon, 1983).

Goals

Motivation influences both attention and maintenance processes (Tennyson & Breuer,

2002), generating the mental effort that drives us to apply our knowledge and skills. Easy goals

are not motivating (Clark, 2003d). Additionally, it has been shown that individuals without

specific goals (such as “do your best”), do not work as long as those with specific goals, such as

“list 70 contemporary authors” (Thompson et al., 2002; Locke & Latham, 2003).

Goal setting theory, according to Thompson et al. (2002), is based on the simple premise

that people exert effort toward accomplishing goals. Goals may increase performance as long as

22


a few factors are taken into account, such as acceptance of the goal, feedback on progress toward

the goal, a goal that is appropriately challenging, and a goal that is specific (Thompson et al.,

2002). Goal setting guides the cognitive strategies in a certain direction. Goal checking are those

monitoring processes that check to see if the goal has been accomplished, or if the selected

strategy is working as expected. The monitoring process is active throughout an activity and

constantly evaluates the success of other processes. If a cognitive strategy appears not to be

working, an alternative may then be selected (Jones et al., 1995).

Goal orientation theory is concerned with the prediction that those with high

performance goals and a perception of high ability will exert great effort, and those with low

ability perceptions will avoid effort (Miller et al., 1996). Once we are committed to a goal, we

must make a plan to achieve the goal. A key element of all goal-directed planning is our personal

assessment of the necessary skills and knowledge required to achieve a goal. A key aspect of

self-efficacy assessment is our perception of how novel and difficult the goal is to achieve. The

ongoing results of this analysis are hypothesized to determine how much effort we will invest in

a goal (Clark, 1999).

Self-Efficacy

A number of items affect motivation and mental effort. In an extensive review of

motivation theories, Eccles and Wigfield (2002) discuss Brokowski and colleagues’ motivation

model that highlights the interaction of the following cognitive, motivational, and self-processes:

knowledge of oneself (including goals and self perceptions); domain-specific knowledge;

strategy knowledge; and personal-motivational states (including attributional beliefs, self-

efficacy, and intrinsic motivation). In a study of college freshmen, Livengood (1992) found that

psychological variables (i.e., effort/ability reasoning, goal choice, and confidence) are strongly

23


associated with academic participation and satisfaction. And Corno and Mandinah (1983)

commented that students in classrooms actively engage in a variety of cognitive interpretations

of their environments and themselves which, in turn, influence the amount and kind of effort

they will expend on classroom tasks.

According to Clark (1999), the more novel the goal is perceived to be, the more effort we

will invest until we believe we might fail. At the point where failure expectations begin, effort is

reduced as we “unchoose” the goal to avoid a loss of control. This inverted U relationship

suggests that mental effort problems include two broad forms: over confidence and under

confidence (Clark, 1999). Therefore, the level of mental effort necessary to achieve goals can be

influenced by adjusting perceptions of goal novelty and the effectiveness of the strategies people

use to achieve goals (Clark, 1999).

Self-efficacy is defined as one’s belief about one’s ability to successfully carry out

particular behaviors (Davis, & Wiedenbeck, 2001). Perceived self-efficacy refers to subjective

judgments of how well one can execute a particular course of action, handle a situation, learn a

new skill or unit of knowledge, and the like (Salomon, 1983). Perceived self-efficacy has much

to do with how a class of stimuli is perceived. The more demanding the stimuli is perceived to

be, the less efficacious the perceiver would feel about it. Conversely, the more familiar, easy, or

shallow it is perceived, the more efficacious the perceiver would feel about handling it. It follows

that perceived self efficacy should be related to the perception of demand characteristics (the

latter includes the perceived worthwhileness of expending effort), and that both should affect

effort investment jointly (Salomon, 1983).

Self-efficacy theory. Self-efficacy theory predicts that students work harder on a learning

task when they judge themselves as capable versus when they lack confidence in their ability to

24


learn. Self-efficacy theory also predicts that students understand the material better when they

have high self-efficacy than when they have low self-efficacy (Mayer, 1998). Effort is primarily

influenced by specific and detailed self efficacy assessments of the knowledge required to

achieve tasks (Clark, 1999). A person’s belief about whether he or she has the skills required to

succeed at a task is possibly the most important factor in the quality and quantity of mental effort

that person will invest (Clark, 2003d).

Expectancy-Value Theory. Related to self-efficacy theories, expectancy-value theories

propose that the probability of behavior depends on the value of the goal and the expectancy of

obtaining that goal (Coffin & MacIntyre, 1999). Expectancies refer to beliefs about how we will

do on different tasks or activities, and values have to do with incentives or reasons for doing the

activity (Eccles & Wigfield, 2002). From the perspective of expectancy-value theory, goal

hierarchies (the importance and the order of goals) also could be organized around aspects of

task value. Different goals may be perceived as more or less useful, or more or less interesting.

Eccles and Wigfield (2002) suggest that the relative value attached to the goal should influence

its placement in a goal hierarchy, as well as the likelihood a person will try to attain the goal and

therefore exert mental effort. Clark (2003b) commented that the more instruction supports a

student’s interest and utility value for instructional goals, as well as the student’s self-efficacy for

the course, the more likely the student will become actively engaged in the instruction and persist

when faced with distractions.

Task value. Task value refers to an individual’s perceptions of how interesting,

important, and useful a task is (Coffin & MacIntyre, 1999). Interest in, and perceived importance

and usefulness of, a task comprise important dimensions of task value (Bong, 2001). Citing

Eccles’ expectancy-value model, Townsend and Hicks (1997) stated that the perception of task

25


value is affected by a number of factors, including the intrinsic value of a task, its perceived

utility value, and its attainment value. Thus, engagement in an academic task may occur because

of interest in the task, or because the task is required for advancement in some other area

(Townsend & Hicks, 1997). According to Corno and Mandinah (1983), a task linked to one’s

aspirations (a “self-relevant” task) is a key condition for task value (Corno & Mandinah, 1983).

Problem-Solving

Problem solving is the intellectual skill to propose solutions to previously unencountered

problem situations (Tennyson & Breuer, 2002). A problem exists when a problem solver has a

goal but does not know how to reach it, so problem solving is mental activity aimed at finding a

solution to a problem (Baker & Mayer, 1999). Problem solving is associated with situations

dealing with previously unencountered problems, requiring the integration of knowledge to form

new knowledge (Tennyson & Breuer, 2002). A first condition of problem solving involves the

differentiation process of selecting knowledge that is currently in storage using known criteria.

Concurrently, this selected knowledge is integrated to form a new knowledge. Cognitive

complexity within this condition focuses on elaborating the existing knowledge base (Tennyson

& Breuer, 2002). Problem solving may also involve situations requiring the construction of

knowledge by employing the entire cognitive system. Therefore, the sophistication of a proposed

solution is a factor of the person’s knowledge base, level of cognitive complexity, higher-order

thinking strategies, and intelligence (Tennyson & Breuer, 2002). According to Mayer (1998),

successful problem solving depends on three components—skill, metaskill, and will—and each

of these components can be influenced by instruction. Metacognition—in the form of metaskill

—is central in problem solving because it manages and coordinates the other components

(Mayer, 1998).

26


O’Neil’s Problem Solving model. O’Neil’s Problem Solving model (O’Neil, 1999, see

figure 1 below) is based on Mayer and Wittrock’s (1996) conceptualization: “Problem solving is

cognitive processing directed at achieving a goal when no solution method is obvious to the

problem solver” (p. 47). This definition is further analyzed into components suggested by the

expertise literature: content understanding or domain knowledge, domain-specific problem-

solving strategies, and self-regulation (see, e.g., O’Neil, 1999, 2002). Self-regulation is

composed of metacognition (planning and self-checking) and motivation (effort and self-

efficacy). Thus, in the specifications for the construct of problem solving, to be a successful

problem solver, “one must know something (content knowledge), possess intellectual tricks

(problem-solving strategies), be able to plan and monitor one’s progress towards solving the

problem (metacognition), and be motivated to perform” (effort and self-efficacy; O’Neil, 1999,

pp. 255-256).

Figure 1. O’Neil’s Problem Solving Model

ContentUnderstanding

Problem-SolvingStrategies

Self-Regulation

DomainSpecific

DomainIndependent

Metacognition Motivation

Planning Self-Monitoring

Effort Self-Efficacy

Problem Solving

27


In problem solving, the skeletal structures are instantiated in content domains, so that a

set of structurally similar models for thinking about problem solving is applied to science,

mathematics, and social studies. These models may vary in the explicitness of problem

representations, the guidance about strategy (if any), the demands of prior knowledge, the focus

on correct procedures, the focus on convergent or divergent responses, and so on (Baker &

Mayer, 1999). Domain-specific aspects of problem solving (e.g., the part that is unique to

geometry, geology, or genealogy) involve the specific content knowledge, the specific

procedural knowledge in the domain, any domain-specific cognitive strategies (e.g., geometric

proof, test, and fix), and domain specific discourse (O’Neil, 1998, as cited in Baker & Mayer,

1999). Both domain-independent and domain-dependent knowledge are usually essential for

problem solving. Domain-dependent analyses focus on the subject matter as the source of all

needed information (Baker & O’Neil, 2002).

Types of Cognitive Load

Cognitive load researchers have identified up to three types of cognitive load. All agree on

intrinsic cognitive load (Brunken et al., 2003; Paas et al., 2003; Renkl, & Atkinson, 2003), which

is the load involved in the process of learning; the load required by metacognition, working

memory, and long-term memory. Another load agreed upon is extraneous cognitive load.

However, it is the scope of this load that is in dispute. To some researchers, any cognitive load

that is not intrinsic cognitive load is extraneous cognitive load. To other researchers, non-

intrinsic cognitive load is divided into germane cognitive load and extraneous cognitive load.

Germane cognitive load is the cognitive load required to process the intrinsic cognitive load

(Renkl, & Atkinson, 2003). From a non-computer-based perspective, this could include

searching a book or organizing notes, in order to process the to-be-learned information. From a

28


computer-based perspective, this could include the interface and controls a learner must interact

with in order to be exposed to, and process, the to-be-learned material. In contrast to germane

cognitive load, these researchers see extraneous cognitive load as the load caused by any

unnecessary stimuli, such as fancy interface designs or extraneous sounds (Brunken et al., 2003).

For each of the two working memory subsystems (visual/spatial, and auditory/verbal), the

total amount of cognitive load for a particular individual under particular conditions is defined as

the sum of intrinsic, extraneous, and germane load induced by the instructional materials.

Therefore, a high cognitive load can be a result of a high intrinsic cognitive load (i.e., the nature

of the instructional content itself). It can, however, also be a result of a high germane cognitive

load (i.e., a result of activities performed on the materials that result in a high memory load) or

high extraneous cognitive load (i.e., a result of inclusion of unnecessary information or stimuli

that result in a high memory load; Brunken et al., 2003).

Low-element interactivity refers to environments where each element can be learned

independently of the other elements, and there is little direct interaction between the elements.

High-element interactivity refers to environments where there is so much interaction between

elements that they cannot be understood until all the elements and their interactions are

processed simultaneously. As a consequence, high-element interactivity material is difficult to

understand (Paas et al., 2003). Element interactivity is the driver of intrinsic cognitive load,

because the demands on working memory capacity imposed by element interactivity are intrinsic

to the material being learned. Reduction in intrinsic load can occur only by dividing the material

into small learning modules (Paas et al., 2003).

Germane or effective cognitive load is influenced by the instructional design. The manner

in which information is presented to learners and the learning activities required of learners are

29


factors relevant to levels of germane cognitive load. Whereas extraneous cognitive load

interferes with learning, germane cognitive load enhances learning (Renkl, & Atkinson, 2003).

Extraneous cognitive load (Renkl, & Atkinson, 2003) is the most controllable load, since it

is caused by materials that are unnecessary to instruction. However, those same materials may be

important for motivation. Unnecessary items are globally referred to as extraneous. However,

another category of extraneous items, seductive details (Mayer, Heiser, & Lonn, 2001), refers to

highly interesting but unimportant elements or instructional segments. These segments usually

contain information that is tangential to the main themes of a story, but are memorable because

they deal with controversial or sensational topics (Schraw, 1998). The seductive detail effect is

the reduction of retention caused by the inclusion of extraneous details (Harp & Mayer, 1998)

and affects both retention and transfer (Moreno & Mayer, 2000).

Complicating the issue of seductive details is the arousal theory which suggests that

adding entertaining auditory adjuncts will make a learning task more interesting, because it

creates a greater level of attention so that more material is processed by the learner (Moreno &

Mayer, 2000). A possible solution is to leave the seductive details, but guide the learner away

from them and to the relevant information (Harp & Mayer, 1998).

While attempting to focus on a mental activity, most of us, at one time or another, have

had our attention drawn to extraneous sounds (Banbury, Macken, Tremblay, & Jones, 2001). On

the surface, seductive details and auditory adjuncts (such as sound effects or music) seem

similar. However, the underlying cognitive mechanisms are quire different. Whereas seductive

details seem to prime inappropriate schemas into which incoming information is assimilated,

auditory adjuncts seem to overload auditory working memory (Moreno & Mayer, 2000).

30


According to Brunken et al. (2003), both extraneous and germane cognitive load can be

manipulated by the instructional design of the learning material (Brunken et al., 2003).

Learner Control

In contrast to more traditional technologies that only deliver information, computerized

learning environments offer greater opportunities for interactivity and learner control. These

environments can offer simple sequencing and pace control, or they can allow the learner to

decide which, and in what order, information will be accessed (Barab, Young, & Wang, 1999).

The term navigation refers to a process of tracking one’s position in an environment, whether

physical or virtual, to arrive at a desire destination. A route through the environment consists of

either a series of locations or a continuous movement along a path (Cutmore et al., 2000).

Effective navigation of a familiar environment depends upon a number of cognitive factors.

These include working memory for recent information, attention to important cues for location,

bearing and motion, and finally, a cognitive representation of the environment which becomes

part of a long-term memory, a cognitive map (Cutmore et al., 2000).

Hypermedia environments divide information into a network of multimedia nodes

connected by various links (Barab, Bowdish, & Lawless, 1997). According to Chalmers (2003),

how easily learners become disoriented in a hypermedia environment may be a function of the

user interface. One area where disorientation can be a problem is in the use of links. Although

links create the advantage of exploration, there is always the chance learners may become lost,

not knowing where they were, where they are going, or where they are (Chalmers, 2003). In a

virtual 3-D environment, Cutmore et al. (2000) argue that navigation becomes problematic when

the whole path cannot be viewed at once and is largely occluded by objects in the environment.

31


Under these conditions, one cannot simply plot a direct visual course from the start to finish

locations. Rather, knowledge of the layout of the space is required (Cutmore et al., 2000).

Message complexity, stimulus features, and additional cognitive demands inherent in

hypermedia, such as learner control, may combine to exceed the cognitive resources of some

learners (Daniels & Moore, 2000). Dillon and Gabbard (1998) found that novice and lower

aptitude students have the greatest difficulty with hypermedia. Children are particularly

susceptible to the cognitive demands of interactive computer environments. According to

Howland, Laffey, and Espinosa (1997), many educators believe that young children do not have

the cognitive capacity to interact and make sense of the symbolic representations of computer

environments.

In spite of the intuitive and theoretical appeal of hypertext environments, empirical

findings yield mixed results with respect to the learning benefits of learner control over program

control of instruction (Niemiec, Sikorski, & Wallberg, 1996; Steinberg, 1989). And six extensive

meta-analyses of distance and media learning studies in the past decade have found the same

negative or weak results (Bernard, et al, 2003). In reference to distance learning environments,

Clark (2003c) argued that when sequencing, contingencies, and learning strategies permit only

minimal learner control over pacing, then “except for the most advanced expert learners, learning

will be increased” (p. 14).

Summary of Cognitive Load

Cognitive Load Theory is based on the assumptions of a limited working memory with

separate channels for auditory and visual/spatial stimuli, and a virtually unlimited capacity long-

term memory that stores schemas of varying complexity and level of automation (Brunken et al.,

2003). According to Paas et al. (2003), cognitive load refers to the amount of load placed on

32


working memory. Cognitive load can be reduced through effective use of the auditory and

visual/spatial channels, as well as schemas stored in long-term memory. Cognitive load also

reflects the measurable concepts of mental load, mental, effort and performance (Paas et al,

2003).

Meaningful learning is defined as deep understanding of the material and is reflected in

the ability to apply what was taught to new situations; i.e., problem solving transfer. (Mayer &

Moreno, 2003). Meaningful learning requires effective metacognitive skills: the management of

cognitive processes (Jones, Farquhar, & Surry, 1995), including selecting relevant information,

organizing connections among the pieces of information, and integrating (i.e., building) external

connections between incoming information and prior knowledge that exists in long-term memory

(Harp & Mayer, 1998). Mental effort refers to the cognitive capacity allocated to a task. Mental

effort is affected by motivation, and motivation cannot exist without goals (Clark, 2003d). Goals

are further affected by self-efficacy, the belief in one’s ability to successfully carry out a

particular behavior (Davis & Wiedenbeck, 2001).

Problem solving is “cognitive processing directed at transforming a given situation into a

desired situation when no obvious methods of solution is available to the problem solver” (Baker

& Mayer, 1999, p. 272). O’Neil’s Problem Solving model (O’Neil, 1999) defines three core

constructs of problem solving: content understanding, problem solving strategies, and self-

regulation. Each of these components is further defined with subcomponents. There are three

types of cognitive load that can be defined in relationship to a learning or problem solving task:

intrinsic cognitive load, germane cognitive load, and extraneous cognitive load.

Intrinsic cognitive load refers to the cognitive load placed on working memory by the to-

be-learned material (Paas et al., 2003). Germane cognitive load refers to the cognitive load

33


required to access and process the intrinsic cognitive load. For example, the problem solving

processes that are instantiated in the learning process so that learning can occur (Renkl &

Atkinson, 2003). Extraneous cognitive load refers to the cognitive load imposed by stimuli that

neither support the learning process (i.e., germane cognitive load) nor are part of the to-be-

learned material (i.e., intrinsic cognitive load). A specific form of extraneous cognitive load is

seductive details; highly interesting but unimportant elements or instructional segment, that are

often used to provide memorable or engaging experiences (Mayer et al., 2001; Schraw, 1998).

An important goal of instructional design is to balance intrinsic, germane, and extraneous

cognitive loads to support learning outcomes, and to recognize that the specific balance is

dependent on a number of factors (Brunen et al., 2003), including the amount of prior knowledge

and the need for motivation.

Learner control, which is inherent in interactive computer-based media, allows for control

of pacing and sequencing (Barab, Young, & Wang, 1999). It also provides an opportunity for

cognitive overload in the form of disorientation; loss of place (Chalmers, 2003). Further, Daniels

and Moore (2000) argued that message complexity, stimulus features, and additional cognitive

demands inherent in hypermedia (e.g., learner control) may combine to exceed the cognitive

resources of some learners. Further, learner control is a potential source for extraneous cognitive

load. Ultimately, these issues may be the cause of mixed reviews of learner control (Bernard, et

al, 2003; Niemiec, Sikorski, & Wallberg, 1996; Steinberg, 1989).

GAMES AND SIMULATIONS

According to Ricci, Salas, and Cannon-Bowers (1996), “computer-based educational

games generally fall into one of two categories: simulation games and video games. Simulation

games model a process or mechanism relating task-relevant input changes to outcomes in a

34


simplified reality that may not have a definite endpoint” (p. 296). Ricci et al. further comment

that simulation games “often depend on learners reaching conclusions through exploration of the

relation between input changes and subsequent outcomes” (p. 296). Video games, on the other

hand, are competitive interactions bound by rules to achieve specified goals that are dependent

on skill or knowledge and often involve chance and imaginary settings (Randel, Morris, Wetzel,

& Whitehill, 1992).

One of the first problems areas with research into games and simulations is terminology.

Many studies that claim to have examined the use of games did not use a game (e.g., Santos,

2002). At best, they used an interactive multimedia that exhibits some of the features of a game,

but not enough features to actually be called a game. A similar problem occurs with simulations.

A large number of research studies use simulations but call them games (e.g., Mayer et al.,

2002). Because the goals and features of games and simulations differ, it is important when

examining the potential effects of the two media to be clear about which one is being examined.

However, there is little consensus in the education and training literature on how games and

simulations are defined.

Games

According to Garris, Ahlers, and Driskell (2002) early work in defining games suggested

that there are no properties that are common to all games and that games belong to the same

semantic category only because they bear a family resemblance to one another. Betz (1995-1996)

argued that a game is being played when the actions of individuals are determined by both their

own actions and the actions of one or more actors.

A number of researchers agree that games have rules (Crookall, Oxford, & Saunders,

1987; Dempsey, Haynes, Lucassen, & Casey, 2002; Garris et al., 2002; Ricci, 1994).

35


Researchers also agree that games have goals and strategies to achieve those goals (Crookall &

Arai, 1995; Crookall et al. 1987; Garris et al., 2002; Ricci, 1994). Many researchers also agree

that games have competition (e.g., Dempsey et al., 2002) and consequences such as winning or

losing (Crookall et al., 1987; Dempsey et al., 2002).

Betz (1995-1996) further argued that games simulate whole systems, not parts, forcing

players to organize and integrate many skills. Students will learn from whole systems by their

individual actions, individual action being the student’s game moves. Crookall et al. (1987) also

noted that a game does not intend to represent any real-world system; it is a “real” system in its

own right. According to Duke (1995), games are situation specific. If well designed for a specific

situation or condition, the same game should not be expected to perform well in a different

environment.

Simulations

In contrast to games, Crookall and Saunders (1989) viewed a simulation as a

representation of some real-world system that can also take on some aspects of reality. Similarly,

Garris et al. (2002) wrote that key features of simulations are they represent real-world systems,

and Henderson, Klemes, and Eshet (2000) commented that a simulation attempts to faithfully

mimic an imaginary or real environment that cannot be experienced directly, for such reasons as

cost, danger, accessibility, or time. Berson (1996) also argued that simulations allow access to

activities that would otherwise be too expensive, dangerous, or impractical for a classroom. Lee

(1999) added that a simulation is defined as a computer program that relates elements together

through cause and effect relationships.

Thiagarajan (1998) argued that simulations do not reflect reality; they reflect someone’s

model of reality. According to Thiagarajan, a simulation is a representation of the features and

36


behaviors of one system through the use of another. At the risk of introducing a bit more

ambiguity, Garris et al. (2002) proposed that simulations can contain game features, which leads

to the final definition: simulation-games.

Simulation-Games

Garris et al. (2002) argued that it is possible to consider games and simulations as similar

in some respects, keeping in mind the key distinction that simulations propose to represent

reality and games do not. Combining the features of the two media, Rosenorn and Kofoed (1998)

described simulation/gaming as a learning environment where participants are actively involved

in experiments, for example, in the form of role-plays, or simulations of daily work situations, or

developmental scenarios.

This paper will use the definitions of games, simulations, and simulation-games as

defined by Gredler (1996), which combine the most common features cited by the various

researchers, and yet provide clear distinctions between the three media. According to Gredler,

Games consist of rules that describe allowable player moves, game constraints

and privileges (such as ways of earning extra turns), and penalties for illegal

(nonpermissable) actions. Further, the rules may be imaginative in that they need

not relate to real-world events (p. 523).

This definition is in contrast to a simulation, which Gredler (1996) defines as “a dynamic

set of relationships among several variables that (1) change over time and (2) reflect authentic

causal processes” (p. 523). In addition, Gredler describes games as linear and simulations as non-

linear, and games as having a goal of winning while simulations have a goal of discovering

causal relationships. Gredler also defines a mixed metaphor referred to as simulation games or

37


gaming simulations, which is a blend of the features of the two interactive media: games and

simulations.

Motivational Aspects of Games

According to Garris et al. (2002), motivated learners are easy to describe; they are

enthusiastic, focused and engaged, they are interested in and enjoy what they are doing, they try

hard, and they persist over time. Furthermore, they are self-determined and driven by their own

volition rather than external forces (Garris et al., 2002). Ricci et al. (1996) defined motivation as

“the direction, intensity, and persistence of attentional effort invested by the trainee toward

training” (p. 297). Similarly, according to Malouf (1987-1988), continuing motivation is defined

as returning to a task or a behavior without apparent external pressure to do so when other

appealing behaviors are available. And more simply, Story and Sullivan (1986) commented that

the most common measure of continuing motivation is whether a student returns to the same task

at a later time.

With regard to video games, Asakawa and Gilbert (2003) argued that, without sources of

motivation, players often lose interest and drop out of a game. However, there seems little

agreement among researchers as to what those sources are—the specific set of elements or

characteristics that lead to motivation in any learning environment, and particularly with

educational games. According to Rieber (1996) and McGrenere (1996), motivational researchers

have offered the following characteristics as common to all intrinsically motivating learning

environments: challenge, curiosity, fantasy, and control (Davis & Wiedenbeck, 2001; Lepper &

Malone, 1987; Malone, 1981; Malone & Lepper, 1987). Malone (1981) and others also included

fun as a criteria for motivation.

38


For interactive games, Stewart (1997) added the motivational importance of goals and

outcomes. Locke and Latham (1990) also commented on the robust findings with regards to

goals and performance outcomes. They argued that clear, specific goals allow the individual to

perceive goal-feedback discrepancies, which are seen as crucial in triggering greater attention

and motivation. Clark (2001) further argued that motivation cannot exist without goals. The

following sections will focus on fantasy, control and manipulation, challenge and complexity,

curiosity, competition, feedback, and fun. The role of goals was discussed previously in this

proposal in fostering effort and motivation was discussed earlier in this document.

Fantasy

Research suggests that material may be learned more readily when presented in an

imagined context that interests the learner than when presented in a generic or decontextualized

form (Garris et al., 2002). Malone and Lepper (1987) defined fantasy as an environment that

evokes “mental images of physical or social situations that do not exist” (p. 250). Rieber (1996)

commented that fantasy is used to encourage learners to imagine they are completing an activity

in a context in which they are really not present. However, Rieber described two types of

fantasies: endogenous and exogenous. Endogenous fantasy weaves relevant fantasy into a game,

while exogenous simply sugar coat a learning environment with fantasy. An example of an

endogenous fantasy would be the use of a laboratory environment to learn chemistry, since this

environment is consistent with the domain. An example of an exogenous environment would be

a using a hangman game to learn spelling, because hanging a person has nothing to do with

spelling. Rieber (1996) noted that endogenous fantasy, not exogenous fantasy, is important to

intrinsic motivation, yet exogenous fantasies are a common and popular element of many

educational games.

39


According to Malone and Lepper (1987), fantasies can offer analogies or metaphors for

real-world processes that allow the user to experience phenomena from varied perspectives. A

number of researchers (Anderson and Pickett, 1978; Ausubal, 1963; Malone and Lepper, 1978;

Malone and Lepper, 1987; Singer, 1973) argued that fantasies in the form of metaphors and

analogies provide learners with better understanding by allowing them to relate new information

to existing knowledge. According to Davis and Wiedenbeck (2001), metaphor also helps learners

to feel directly involved with objects in the domain so the computer and interface become

invisible.

Control and Manipulation

Hannifin and Sullivan (1996) define control as the exercise of authority or the ability to

regulate, direct, or command something. Control, or self-determination, promotes intrinsic

motivation because learners are given a sense of control over the choices of actions they may

take (deCharms, 1986; Deci, 1975; Lepper & Greene, 1978). Furthermore, control implies that

outcomes depend on learners’ choices and, therefore, learners should be able to produce

significant effects through their own actions (Davis, & Wiedenbeck, 2001). According to Garris

et al. (2002), games evoke a sense of personal control when users are allowed to select strategies,

manage the direction of activities, and make decisions that directly affect outcomes, even if those

actions are not instructionally relevant.

However, Hannafin & Sullivan (1996) warned that research comparing the effects of

instructional programs that control all elements of the instruction (program control) and

instructional programs in which the learner has control over elements of the instructional

program (learner control) on learning achievement has yielded mixed results. Dillon and

Gabbard (1998) commented that novice and lower aptitude students have greater difficulty when

40


given control, compared to experts and higher aptitude students, and Niemiec, Sikorski, and

Walberg (1996) argued that control does not appear to offer any special benefits for any type of

learning or under any type of condition.

Challenge and complexity

Challenge, also referred to as effectance, competence, or mastery motivation (Bandura,

1977; Csikszentmihalyi, 1975; Deci, 1975; Harter, 1978; White, 1959), embodies the idea that

intrinsic motivation occurs when there is a match between a task and the learner’s skills. The

task should not be too easy or too hard, because in either case, the learner will lose interest

(Clark, 1999; Malone & Lepper, 1987). Clark (1999) describes this effect as an inverted U-

shaped relationship with lack of effort existing on either side of difficultly ranging from too easy

to too hard. Stewart (1997) similarly commented that games that are too easy will be dismissed

quickly. According to Garris et al. (2002), there are several ways in which an optimal level of

challenge can be obtained. Goals should be clearly specified, yet the probability of obtaining that

goal should be uncertain, and goals must also be meaningful to the individual. Garris and

colleagues argued that linking activities to valued personal competencies, embedding activities

within absorbing fantasy scenarios, or engaging competitive or cooperative motivations could

serve to make goals meaningful.

Curiosity

According to Rieber (1996), challenge and curiosity are intertwined. Curiosity arises from

situations in which there is complexity, incongruity, and discrepancy (Davis & Wiedenbeck,

2001). Sensory curiosity is the interest evoked by novel situations and cognitive curiosity is

evoked by the desire for knowledge (Garris et al. 2002). Cognitive curiosity motivates the learner

to attempt to resolve the inconsistency through exploration (Davis, & Wiedenbeck, 2001).

41


Curiosity is identified in games by unusual visual or auditory effects, and by paradoxes,

incompleteness, and potential simplifications (Westbrook & Braithwaite, 2002). Curiosity is the

desire to acquire more information, which is a primary component of the players’ motivation to

learn how to operate the game (Westbrook & Braithwaite, 2001).

Malone and Lepper (1987) noted that curiosity is one of the primary factors that drive

learning and is related to the concept of mystery. Garris et al. (2002) commented that curiosity is

internal, residing in the individual, and mystery is an external feature of the game itself. Thus,

mystery evokes curiosity in the individual, and this leads to the question of what constitutes

mystery (Garris et al. 2002). Research suggests that mystery is enhanced by incongruity of

information, complexity, novelty, surprise, and violation of expectations (Berlyne, 1960),

incompatibility between ideas and inability to predict the future (Kagan, 1972), and information

that is incomplete and inconsistent (Malone & Lepper, 1987).

Competition

Studies on competition with games and simulations have mixed results, due to

preferences and reward structures. A study by Porter, Bird, and Wunder (1990-1991) examining

competition and reward structures found that the greatest effects of reward structure were seen in

the performance of those with the most pronounced attitudes toward either competition or

cooperation. The results also suggested that performance was better when the reward structure

matched the individual’s preference. According to the authors, implications are that emphasis on

competition will enhance the performance of some learners but will inhibit the performance of

others (Porter et al., 1990-1991).

Yu (2001) investigated the relative effectiveness of cooperation with and without inter-

group competition in promoting student performance, attitudes, and perceptions toward subject

matter studied, computers, and interpersonal context. With fifth-graders as participants, Yu

42


found that cooperation without inter-group competition resulted in better attitudes toward the

subject matter studies and promoted more positive inter-personal relationships both within and

among the learning groups, as compared to competition (Yu, 2001). The exchange of ideas and

information both within and among the learning groups also tended to be more effective and

efficient when cooperation did not take place in the context of inter-group competition (Yu,

2001).

Feedback

Feedback within games can be provided for learners to quickly evaluate their progress

against the established game goal. This feedback can take many forms, such as textual, visual,

and aural (Rieber, 1996). According to Ricci et al. (1996), within the computer-based game

environment, feedback is provided in various forms including audio cues, score, and remediation

immediately following performance. The researchers argued that these feedback attributes can

produce significant differences in learner attitudes, resulting in increased attention to the learning

environment. Clark (2003a) argued that, for feedback to be effective, it must be based on

“concrete learning goals that are clearly understood” (p. 18) and that it describe the gap between

the learner’s current performance and the goal. Additionally, the feedback must not be focused

on the failure to achieve the goal (Clark, 2003a).

Fun

Quinn (1994, 1997) argued that for games to benefit educational practice and learning,

they need to combine fun elements with aspects of instructional design and system design that

include motivational, learning, and interactive components. According to Malone (1981) three

elements (fantasy, curiosity, and challenge) contribute to the fun in games. While fun has been

cited as important for motivation and, ultimately, for learning, there is little empirical evidence

43


supporting the concept of fun. It is possible that fun is not a construct but, rather, represents other

concepts or constructs. Relevant alternative concepts or constructs include play, engagement, and

flow.

Play is entertainment without fear of present or future consequences; it is fun (Resnick &

Sherer, 1994). According to Rieber, Smith, and Noah (1998), play describes an intense learning

experience in which both adults and children voluntarily devote enormous amounts of time,

energy, and commitment and, at the same time, derive great enjoyment from the experience; this

is termed serious play (Rieber et al., 1998). Webster et al. (1993) found that labeling software

training as play showed improved motivation and performance. According to Rieber and Matzko

(2001), serious play is an example of an optimal life experience.

Csikszentmihalyi (1975; 1990) defines an optimal experience as one in which a person is

so involved in an activity that nothing else seems to matter; termed flow or a flow experience.

When completely absorbed in and activity, he or she is ‘carried by the flow,’ hence the origin of

the theory’s name (Rieber and Matzko, 2001). Rieber and Matzko (2001) offered a broader

definition of flow, commenting that a person may be considered in flow during an activity when

experiencing one or more of the following characteristics: Hours pass with little notice;

challenge is optimized; feelings of self-consciousness disappear; the activity’s goals and

feedback are clear; attention is completely absorbed in the activity; one feels in control; and one

feels freed from other worries (Rieber & Matzko, 2001). According to Davis and Wiedenbeck

(2001), an activity that is highly intrinsically motivating can become all-encompassing to the

extent that the individual experiences a sense of total involvement, losing track of time, space,

and other events. Davis and Wiedenbeck also argued that the interaction style of a software

package is expected to have a significant effect on intensity of flow. It should be noted that

44


Rieber and Matzko (2001) have contended that play and flow differ in one respect; learning is an

expressed outcome of serious play but not of flow.

Engagement is defined as a feeling of directly working on the objects of interest in the

worlds rather than on surrogates. According to Davis and Wiedenbeck (2001), this interaction or

engagement can be used along with the components of Malone and Lepper’s (1987) intrinsic

motivation model to explain the effect of an interaction style on intrinsic motivation, or flow.

Garris et al. (2002) commented that training professionals are interested in the intensity of

involvement and engagement that computer games can invoke, to harness the motivational

properties of computer games to enhance learning and accomplish instructional objectives.

Learning and Other Outcomes for Games

Results from studies reporting on the performance and learning outcomes from games are

mixed. This section is subdivided into four discussions. First will be a discussion of studies

indicating positive results regarding performance and learning outcomes attributed to games and

simulations. Second will be a discussion of studies indicating negative or null results regarding

performance and learning outcomes attributed to games and simulations. Third will be a

discussion of the relationship of instructional design to effectiveness of educational games and

simulations as an explanation of mixed results from game and simulation studies. Last will be a

discussion of reflection and debriefing as a necessary component to learning, with specific

references to the learning instantiated in games and simulations.

Positive Outcomes from Games and Simulations

Simulations and games have been cited as beneficial for a number of disciplines and for a

number of educational and training situations, including aviation training (Salas, Bowers, &

Rhodenizer, 1998), aviation crew resource management (Baker, Prince, Shrestha, Oser, & Salas,

45


1993), military mission preparation (Spiker & Nullmeyer, n.d.), laboratory simulation (Betz,

1995-1996), chemistry and physics education (Khoo & Koh, 1998), urban geography and

planning (Adams, 1998; Betz, 1995-1996), farm and ranch management (Cross, 1993), language

training (Hubbard, 1991), disaster management (Stolk, Alexandrian, Gros, & Paggio, 2001), and

medicine and health care (Westbrook & Braithwaite, 2001; Yair, Mintz, & Litvak, 2001). For

business, games and simulations have been cited as useful for teaching strategic planning

(Washburn & Gosen, 2001; Wolfe & Roge, 1997), finance (Santos, 2002), portfolio management

(Brozik, & Zapalska, 2002), marketing (Washburn & Gosen), knowledge management

(Leemkuil, de Jong, de Hoog, & Christoph, 2003), and media buying (King & Morrison, 1998).

In addition to teaching domain-specific skills, games have been used to impart more

generalizable skills. Since the mid 1980s, a number of researchers have used the game Space

Fortress, a 2-D, simplistic arcade-style game, with a hexagonal “fortress” in the center of the

screen surrounded by two concentric hexagons, and a space ship, to improve spatial and motor

skills that transfer far outside gameplay, such as significantly improving the results of fighter

pilot training (Day, Arthur, and Gettman, 2001). Also, in a series of five experiments, Green and

Bavelier (2003) showed the potential of video games to significantly alter visual selection

attention. Similarly, Greenfield, DeWinstanley, Kilpatrick, & Kaye (1994) found, with

experiments involving college students, that video game practice could significantly alter the

participants’ strategies of spatial attentional deployment.

According to Ricci et al. (1996), results of their study provided evidence that computer-

based gaming can enhance learning and retention of knowledge. They further commented that

positive trainee reaction might increase the likelihood of student involvement with training (i.e.,

devote extra time to training). In a more guarded position, Leemkuil et al. (2003) commented

46


that much of the work on the evaluation of games has been anecdotal, descriptive, or judgmental,

yet there are some indications that they are effective and superior to case studies in producing

knowledge gains, especially in the area of strategic management (Wolfe, 1997).

Negative or Null Outcomes from Games and Simulations

A number of researchers have addressed the issue of the motivational aspects of games,

arguing that the motivation attributed to enjoyment of educational games may not necessarily

indicate learning and, possibly, might indicate less learning. Garris et al. (2002) noted that,

although students generally seem to prefer games over other, more traditional, classroom training

media, reviews have reported mixed results regarding the training effectiveness of games.

Druckman (1995) concluded that games seem to be effective in enhancing motivation and

increasing student interest in subject matter, yet the extent to which that translates into more

effective learning is less clear. With caution, Brougere (1999) commented that anything that

contributes to the increase of emotion (such as the quality of the design of video games)

reinforces the attraction of the game but not necessarily its educational interest. Similarly, Salas

et al. (1998) commented that liking a simulation does not necessarily transfer to learning.

Salomon (1984) went even further, by commenting that a more positive attitude can actually

indicate less learning. And in an early meta-analysis of the effectiveness of simulation games,

Dekkers and Donatti (1981) found a negative relationship between duration of training and

training effectiveness. Simulation games became less effective the longer the game was used

(suggesting that perhaps trainees became bored over time).

Relationship of Instructional Design to Effective Games and Simulations

de Jong and van Joolingen (1998), after reviewing a large number of studies on learning

from simulations, concluded, “there is no clear and univocal outcome in favor of simulations. An

47


explanation why simulation based learning does not improve learning results can be found in the

intrinsic problems that learners may have with discovering learning” (p. 181). These problems

are related to processes such as hypothesis generation, design of experiments, interpretation of

data, and regulation of learning. After analyzing a large number of studies, de Jong and van

Joolingen (1998) concluded that adding instructional support to simulations might help to

improve the situation.

The generally accepted position is that games themselves are not sufficient for learning but

there are elements in games that can be activated within an instructional context that may

enhance the learning process (Garris et al., 2002). In other words, outcomes are affected by the

instructional strategies employed (Wolfe, 1997). Leemkuil et al. (2003), too, commented that

there is general consensus that learning with interactive environments such as games,

simulations, and adventures is not effective when no instructional measure or support is added.

According to Thiagarajan (1998), if not embedded with sound instructional design, games

and simulations often end up truncated exercises often mislabeled as simulations. Gredler (1996)

further commented that poorly developed exercises are not effective in achieving the objectives

for which simulations are most appropriate—that of developing students’ problem-solving skills.

Lee (1999) commented that effect size never tells us under what conditions students learn more,

less, or not at all compared with the comparison group. For instructional prescription, we need

information dealing with instructional variables, such as instructional mode, instructional

sequence, knowledge domain, and learner characteristics (Lee, 1999).

48


Reflection and Debriefing

Instructional strategies that researchers have suggested as beneficial to learning from

games and simulations are reflection and debriefing. Brougere (1999) argued that a game cannot

be designed to directly provide learning. A moment of reflexivity is required to make transfer

and learning possible. Games require reflection, which enables the shift from play to learning.

Therefore, debriefing (or after action review), which includes reflection, appears to be an

essential contribution to research on play and gaming in education (Brougere, 1999; Leemkuil et

al., 2003; Thiagarajan, 1998). According to Garris et al. (2002), debriefing is the review and

analysis of events that occurred in the game. Debriefing provides a link between what is

represented in the simulation or gaming experience and the real world. It allows the learners to

draw parallels between game events and real-world events. Debriefing allows learners to

transform game events into learning experiences. Debriefing may include a description of events

that occurred in the game, analysis of why they occurred, and the discussion of mistakes and

corrective actions. Garris et al. (2002) argued that learning by doing must be coupled with the

opportunity to reflect and abstract relevant information for effective learning to occur.

Summary of games and simulation section.

Computer-based educational games fall into three categories: games, simulations, and

simulation games. Games consist of rules, can contain imaginative contexts, are primarily linear,

and include goals as well as competition, either against other players or against a computer

(Gredler, 1996). Simulations display the dynamic relationship among variables which change

over time and reflect authentic causal processes. Simulations are non-linear and have a goal of

discovering causal relationships through manipulation of independent variables. Simulation

games are a blend of games and simulations (Gredler, 1996).

49


Clark (2003d) argued that mental effort is affected by motivation. Beginning with the

work of Malone (1981), a number of constructs have been described as providing the

motivational aspects of games: fantasy, control and manipulation, challenge and complexity,

curiosity, competition, feedback, and fun. Fantasy is defined as an environment that evokes

“mental images of physical or social situations that do not exist” (Malone & Lepper, 1987, p.

250). Malone & Lepper (1987) also commented that fantasies can offer analogies and metaphors,

and Davis and Wiedenbeck (2001) argued that metaphors can help learners feel more directly

involved in the domain.

Control and manipulation promote intrinsic motivation, because learners are given a

sense of control over their choices and actions (deCharms, 1986, Deci, 1975). Challenge

embodies the idea that intrinsic motivation occurs when there is a match between a task and the

learner’s skills (Bandura, 1977, Csikszentmihalyi, 1975; Harter, 1978). The task should be

neither too hard nor too easy, otherwise, in both cases, the learner would lose interest (Clark,

1999; Malone & Lepper, 1987). According to Rieber (1996), curiosity and challenge are

intertwined. Curiosity arises from situations in which there is complexity, incongruity, and

discrepancy (Davis & Wiedenbeck, 2001) and Malone and Lepper (1997) argued that curiosity is

one of the primary factors that drive learning.

While Malone (1981) defines competition as important to motivation, studies on

competition with games and simulations have resulted in mixed findings, due to individual

learner preferences, as well as the types of reward structures connected to the competition (e.g.,

Porter, Bird, & Wunder, 1990-1991; Yu, 2001). Another motivational factor in games, feedback,

allows learners to quickly evaluate their progress and can take many forms, such as textual,

visual, and aural (Rieber, 1996). Ricci et al. (1996) argued that feedback can produce significant

50


differences in learner attitudes, resulting in increased attention to a learning environment.

However, Clark (2003a) commented that feedback must be focused on clear learning goals and

current performance results.

The last category contributing to motivation, fun, is possibly an erroneous category. Little

empirical evidence exists for the construct. However, evidence does support the related

constructs of play, engagement, and flow. Play is entertainment without fear of present of future

consequences (Resnick & Sherer, 1994). Webster et al. (1993) found that labeling software

training as play improved motivation and performance. Csikszentmihalyi (1975; 1990) defines

flow as an optimal experience in which a person is so involved in an activity that nothing else

seems to matter. According to Davis and Wiedenbeck (2001), engagement is the feeling of

working directly on the objects of interest in a world, and Garris et al. (2002) argued that

engagement can harness the motivational properties of computer games to enhance learning and

accomplish instructional objectives.

While numerous studies have cited the learning benefits of games and simulations (e.g.,

Adams, 1998; Baker et al., 1997; Betz, 1995-1996; Khoo & Koh, 1998), others have found

mixed, negative, or null outcomes from games and simulations, specifically in relationship to the

of enjoyment of a game to learning from the game (e.g., Brougere, 1999; Dekkers & Donatti,

1981; Druckman, 1995). There appears to be consensus among a large number of researcher with

regards to the negative, mixed, or null findings, suggesting that the cause might be a lack of

sound instructional design embedded in the games (de Jong & van Joolingen, 1998; Garris et al.,

2002; Gredler, 1996; Lee, 1999; Leemkuil et al., 2003; Thiagarajan, 1998; Wolfe, 1997).

Among the various instructional strategies, reflection and debriefing have been cited as

critical to learning with games and simulations. Brougere (1999) argued that games cannot be

51


designed to directly provide learning; reflection is required to make transfer and learning

possible. Debriefing provides an opportunity for reflection (Brougere, 1999, Garris et al., 2002;

Thiagarajan, 1998).

ASSESSMENT OF PROBLEM SOLVING

According to O’Neil’s Problem Solving model (1999), successful problem solving requires

content understanding, problem solving strategies, and self-regulation. Therefore, proper

assessment of problem solving should address all three constructs.

Measurement of Content Understanding

According to Mayer and Moreno (2003), meaningful learning is reflected in retention and

transfer. Transfer refers to the ability to apply what was taught to new situations. According to

Day et al. (2001), declarative knowledge, which is an indication of retention, reflects the amount

of knowledge or facts learned. Similarly, Davis and Wiedenbeck (20010 commented that

meaningful learning results in an understanding of the basic concepts of the new material

through its integration with existing knowledge. Mayer and Moreno (1998) assessed content

understanding with retention and transfer questions. In their study on the split-attention effect in

multimedia learning, they used a retention test and a matching test containing a series of

questions to assess the extent to which participants retained knowledge delivered by the

multimedia.

Day et al. (2001), proposed knowledge maps as an alternative method to measure content

understanding. A knowledge map is a structural representation that consists of nodes and links.

Each node represents a concept in the domain of knowledge. Each link, which connects two

nodes, represents the relationship between the nodes; that is, the relationship between the two

concepts (Schau & Mattern, 1997). Knowledge structures are based on the premise that people

52


organize information into patterns that reflect the relationships which exist between concepts and

the features that define them (Day et al., 2001). Day et al. further argued that, in contrast to

declarative knowledge which reflects the amount of knowledge or facts learned, knowledge

structures represent the organization of the knowledge.

As Schau and Mattern (1997) point out, learners should not only be aware of the concepts

but also of the connections among them. In a training context, knowledge structures reflect the

degree to which trainees have organized and comprehended the content of training (Day et al.,

2001). Knowledge maps, which are graphical representations of knowledge structures, have been

used as an effective tool to learn complex subjects (Herl et al., 1996) and to facilitate critical

thinking and (West, Pomeroy, Park, Gerstenberger, & Sandoval, 2000). Several studies also

revealed that knowledge maps are not only useful for learning, but are a reliable and efficient

measurement of content understanding, as well (Herl et al., 1999; Ruiz-Primo, Schultz, &

Shavelson, 1997). The results of a study by Day et al. (2001) indicated that knowledge

structures are predictive of both skill retention and skill transfer and can therefore be a viable

indices of training outcomes, and Ruiz-Primo et al. (1997) proposed a framework for

conceptualizing knowledge maps as a potential assessment tool in science, because it allows for

organization and discrimination between concepts.

Ruiz-Primo et al. (1997) claimed that as an assessment tool, knowledge maps are

identified as a combination of three components: (a) a task that allows a student to perform his or

her content understanding in the specific domain (b) a format for student’s responses, and (c) a

scoring system by which the student’s knowledge map could be accurately evaluated. Chuang

(2003) modified this framework to serve as an assessment specification using a concept map.

Researchers have successfully applied knowledge maps to measure students’ content

53


understanding in science for both high school students and adults (e.g., Chuang, 2003; Herl et al.,

1999; Schacter et al., 1999; Schau et al., 2001). For example, Schau et al. (2001) used select-and-

fill-in knowledge maps to measure secondary and postsecondary students’ content understanding

of science in two studies. The results of the participant’s performance on the knowledge maps

correlated significantly with that of a multiple choice test, a traditional measure of learning

(r= .77 for eighth grade and r=. 74 for seventh grade), providing validity to the use of knowledge

maps to assess learning outcomes. CRESST developed a computer-based knowledge mapping

system, which measures the deeper understanding of individual students and teams, reflects

thinking processes in real-time, and economically reports student thinking process data back to

teachers and student (Chung et al., 1999; O’Neil, 1999; Schacter et al., 1999). The computer-

based knowledge map has been used successfully in a number of studies (e.g., Chuang, 2003;

Chung et al., 1999; Hsieh, 2001; Schacter et al., 1999).

In the four studies, the map contained 18 concepts of environmental science, and seven

links for relationships, such as “cause,” “influence,” and “used for.” Subjects were asked to

create a knowledge map in the computer-based environment. In the study conducted by Schacter

et al. (1999) students were evaluated by creating individual knowledge map, after searching a

simulated World Wide Web environment. In the studies conducted by Chung et al. (1999), Hsieh

(2001), and Chuang (2003) two students constructed a group map cooperatively through

networked computers. Results of the cooperative studies showed that using networked computers

to measure group processes was feasible. Figures 2 and 3 show the knowledge map used for the

three studies.

54


Figure 2: User Interface

As seen in Figure 2, the screen of computer was divided into three major parts. The bottom

section was for communication between the two team members. The middle section is where the

one of the team members constructed the knowledge map. The top section contains four menu

items: “Session,” “Add Concept,” “Available Links,” and “About.” Figure three shows the drop-

down menu that appears when “Add Concept” is clicked. Clicking when the mouse pointer is

over a concept adds that link to the knowledge map. Figure 2 shows three concepts that were

added: desk, safe, and key. Figure 2 also shows links that were added to the concept map by (A)

clicking on one concept on the screen, holding the shift-key down and clicking a second concept,

so that both concepts are “selected,” then clicking on the “Available Links” menu and selection

the appropriate link type from the drop-down menu that appeared.

55


Figure 3: Adding Concepts to the Knowledge Map

Measurement of Problem Solving Strategies

According to Baker and Mayer (1999), “Problem solving is cognitive processing directed

at transforming a given situation into a desired situation when no obvious method of solution is

available to the problem solver” (p. 272). Simply put, problem solving is mental activity aimed

at finding a solution to a problem. According to Baker and Maker, a promising direct approach

to knowledge representation, “more parsimonious than a traditional performance assessment,” is

knowledge or concept mapping, in which “the learner constructs a network consisting of nodes

(e.g., key words or terms) and links (e.g., ‘is part of’, ‘led to’, ‘is an example of’)” (p. 274).

Problem solving strategies can be categorized as domain-independent (-general) and

domain-dependent (-specific) (Alexander, 1992; Bruning, Schraw, & Ronning, 1999; O’Neil,

1999; Perkins & Salomon, 1989). Domain-specific knowledge is knowledge about a particular

56


field of study or a subject, such as the application of equations in a math question, the application

of a formula in a chemistry problem, or the specific strategies to be successful in a game.

Domain-general knowledge is the broad array of problem solving knowledge that is not linked

with a specific domain, such as the application of multiple representations and analogies in a

problem-solving task or the use of Boolean search strategies in a search task (Chuang, 2003).

Transfer questions have been examined as an alternative to performing transfer tasks. For

example, in a recent study, Mayer and Moreno (1998) assessed participants’ problem-solving

strategies with a list of transfer questions. Responses to the transfer questions were positively

correlated with performance as measured by retention and transfer, indicating that transfer

questions are a viable alternative to transitions methods of measuring retention and transfer, such

as tests and novel problem solving (Mayer & Moreno, 1998).

Measure of Self-Regulation

While Brunning, Schraw, and Ronning (1999), commented that some researchers believe

self-regulation includes three core components—metacognitive awareness, strategy use, and

motivational control—, according to O’Neil’s Problem Solving model (O’Neil, 1999), self-

regulation is composed of only two core components: metacognition and motivation. Strategy

use is a separate construct that encompasses domain-specific and domain-general knowledge.

Within O’Neil’s model, metacognition encompasses two subcategories, planning and self-

checking/monitoring (Hong & O’Neil, 2001; O’Neil & Herl, 1998; Pintrich & DeGroot, 1990),

and motivation is indicated by mental effort and self-efficacy (Zimmerman, 1994; 2000).

O’Neil and Herl (1998) developed a trait self-regulation questionnaire examining the four

components of self-regulation (planning, self-checking/monitoring, mental effort, and self-

efficacy). Of the four components, planning is the first step, since learners must have a plan to

57


achieve the proposed goal. Self-efficacy is one’s belief in his or her capability to accomplish a

task, and effort is amount of mental effort exerted on a task. In the trait self-regulation

questionnaire developed by O’Neil and Herl, planning, self-checking, self-efficacy, and effort

are assessed using eight questions each. The reliability of this self-regulation inventory has been

established in previous studies (e.g., Hong & O’Neil, 2001). For example, in the research

conducted by Hong and O’Neil (2001), the reliability estimates (coefficient α) of the four

subscales of self-regulation, planning, self-checking, effort, and self-efficacy were .76, .06, .83,

and .85 respectively; Research also provided evidence for construct validation. To evaluate

problem-solving ability, previous researchers (e.g., Baker & Mayer, 1999; Baker & O’Neil,

2002; Mayer, 2002; O’Neil, 1999) argued that computer-based assessment has the merit of

integrating validity to generate test items and the efficiency of computer technology as a means

of presenting and scoring tests.

Summary of Problem Solving Assessment

Problem solving is cognitive processing directed at achieving a goal when no solution

method is obvious to the problem solver (Baker & Mayer, 1999). In O’Neil’s Problem Solving

model (O’Neil, 1999), problem solving strategies can be categorized into two types: domain-

independent (-general) and domain-dependent (-specific) problem solving strategies. Self-

regulation includes two sub-categories: metacognition and motivation. Metacognition is

composed of self-checking/monitoring and planning. Motivation is comprised of effort and self-

efficacy.

Knowledge maps are reliable and efficient for the measurement of content understanding.

CRESST has developed a simulated Internet Web space to evaluate problem solving strategies

such as information searching strategies and feedback inquiring strategies. Computer-based

58


problem-solving assessments are economical, efficient and valid measures that employ

contextualized problems that require students to think for extended periods of time and to

indicate the problem-solving heuristics they were using and why.

SCAFFOLDING

As discussed in earlier, cognitive load theory is concerned with methods for reducing the

amount of cognitive load placed on working memory during learning and problem solving

activities. Clark (2003b) commented that instructional methods must also keep the cognitive load

from instructional presentations to a minimum. Scaffolding is considered a viable instructional

method that assists in cognitive load reduction. There are a number of definitions of scaffolding

in the literature. Chalmers (2003) defines scaffolding as the process of forming and building

upon a schema (Chalmers, 2003). In a related definition, van Merrionboer, Kirshner, and Kester

(2003) defined the original meaning of scaffolding as all devices or strategies that support

students’ learning. More recently, van Merrienboer, Clark, and de Croock (2002) defined

scaffolding as the process diminishing support as learners acquire more expertise. Allen (1997)

defined scaffolding as the process of training a student on core concepts and then gradually

expanding the training. Ultimately, the core principle embodied in each of these definitions is

that scaffolding is concerned with the amount of cognitive load imposed by learning and

provides methods for reducing that load. Therefore, for the purposes of this review, all four

definitions of scaffolding will be considered.

As defined by Clark (2001), instructional methods are external representations of internal

cognitive processes that are necessary for learning but which learners cannot or will not provide

for themselves. They provide learning goals (e.g., demonstrations, simulations, and analogies:

Alessi, 2000; Clark 2001), monitoring (e.g., practice exercises: Clark, 2001), feedback (Alessi,

59


2000; Clark 2001; Leemkuil, de Jong, de Hoog, & Christoph, 2003), and selection (e.g.,

highlighting information: Alessi, 2000; Clark, 2001). In addition, Alessi (2000) adds: giving

hints and prompts before student actions; providing coaching, advice, or help systems; and

providing dictionaries and glossaries. Jones et al. (1995) added advance organizers, graphical

representations of problems, and hierarchical knowledge structures. Each of these examples is a

form of scaffolding.

In learning by doing in a virtual environment, students can actively work in realistic

situations that simulate authentic tasks for a particular domain (Mayer et al., 2002). A major

instructional issue in learning by doing within simulated environments concerns the proper type

of guidance, that is, how best to create cognitive apprenticeship (Mayer et al. 2002). Mayer et al.

(2002) also commented that their research shows that discovery-based learning environments can

be converted into productive venues for learning when appropriate cognitive scaffolding is

provided; specifically, when the nature of the scaffolding is aligned with the nature of the task,

such as pictorial scaffolding for pictorially-based tasks and textual-based scaffolding for

textually-based tasks. For example, in a recent study, Mayer et al. (2002) found that students

learned better from a computer-based geology simulation when they were given some support

about how to visualize geological features, versus textual or auditory guidance.

Graphical Scaffolding

According to Allen (1997), selection of appropriate text and graphics can aid the

development of mental models, and Jones et al. (1995) commented that visual cues such as maps

and menus as advance organizers help learners conceptualize the organization of the information

in a program (Jones et al., 1995). A number of researchers support the use of maps as visual aids

60


and organizers (Benbasat & Todd, 1993: Chou & Lin, 1998; Chou, Lin, & Sun, 2000; Farrell &

Moore, 2000-2001; Ruddle et al, 1999)

Chalmers (2003) defines graphic organizers is organizers of information in a graphic

format, which act as spatial displays of information that can also act as study aids. Jones et al.

(1995) argued that interactive designers should provide users with visual or verbal cues to help

them navigate through unfamiliar territory. Overviews, menus, icons, or other interface design

elements within the program should serve as advance organizers for information contained in the

interactive program (Jones et al., 1995). In addition, the existence of bookmarks enables

recovering from the possibility of disorientation; loss of place (Dias, Gomes, & Correia, 1999).

However, providing such support devices does not guarantee learners will use them. For

example, in an experiment involving a virtual maze, Cutmore et al. (2000) found that, while

landmarks provided useful cues, males utilized them significantly more often than females did.

Navigation maps

Cutmore et al. (2000) define navigation as “…a process of tracking one’s position in a

physical environment to arrive at a desired destination” (p. 224). A route through the

environment consists of either a series of locations or a continuous movement along a path.

Cutmore et al. further commented that “Navigation becomes problematic when the whole path

cannot be viewed at once but is largely occluded by objects in the environment’” (p. 224). These

may include internal walls or large environmental features such as trees, hills, or buildings.

Under these conditions, one cannot simply plot a direct visual course from the start to finish

locations. Rather, knowledge of the layout of the space is required. Navigation maps or other

descriptive information may provide that knowledge (Cutmore et al. 2000).

61


Effective navigation of a familiar environment depends upon a number of cognitive

factors. These include working memory for recent information, attention to important cues for

location, bearing and motion, and finally, a cognitive representation of the environment which

becomes part of a long-term memory, a cognitive map (Cutmore et al., 2000). According to Yair,

Mintz, and Litvak (2001), the loss of orientation and “vertigo” feeling which often accompanies

learning in a virtual-environment is minimized by the display of a traditional, two-dimensional

dynamic map. The map helps to navigate and to orient the user, and facilitates an easier learning

experience. Dempsey et al. (2002) also commented that an overview of player position was

considered an important feature in adventure games.

A number of experiments have examined the use of navigation maps in virtual

environments. Chou and Lin (1998) and Chou et al. (2000) examined various navigation map

types, with some navigation maps offering global views of the environment and others offering

more localized views, based on the learner’s current location. In their experiments using over one

hundred college students, they found that any form of navigation map produced more efficient

navigation of the site as well as better development of cognitive maps (concept or knowledge

maps), compared to having no navigation map. Additionally, the global navigation map results

for navigation and concept map creation were significantly better than any of the local navigation

map variations or the lack of navigation map, while use of the local navigation maps was not

significantly better than not having a navigation map. This suggests that, while the use of

navigation maps is helpful, the nature or scope of the navigation map influences its effectiveness

(Chou & Lin, 1998).

According to Tkacz (1998), soldiers use navigation maps as tools, which involves spatial

reasoning, complex decision making, symbol interpretation, and spatial problem solving. In her

62


study, Tkacz (1998) examined the procedural components of cognitive maps required for using

and understanding topographic navigation maps, stating that navigation map interpretation

involves both top-down (retrieved from long-term memory) and bottom-up (retrieved from the

environment and the navigation map) procedures. Therefore, Tkacz examined the cognitive

components underlying navigation map interpretation to assess the influence of individual

differences on course success and on real world position location. In addition, Tkacz, related

position location ability to video game performance in a simulated environment. The learning

goal of the study was for 105 marines (non-random assignment) to learn to match their position

in the real world with a navigation map. From the results of the study, Tkacz argued that spatial

orientation is the most important cognitive component of terrain association and that orientation

and, to a lesser extent, reasoning ability are important in navigation map interpretation and

course performance. It should be noted that, while Tkacz referred to the instrument as a game, it

appears to fit the Gredler’s (1996) definition of a simulation, not a game or simulation game.

Mayer et al. (2002) commented that a major instructional issue in learning by doing

within simulated environments concerns the proper type of guidance, which they refer to as

cognitive apprenticeship. The investigators used a geological gaming simulation, the Profile

Game, to test various types of guidance structures (i.e., strategy modeling), ranging from no

guidance to illustrations (i.e., pictorial aids) to verbal descriptions to pictorial and verbal aids

combined. The Profile Game is based on the premise, “Suppose you were visiting a planet and

you wanted to determine which geological feature is present on a certain portion of the planet’s

surface” (Mayer et al., p. 171). While exploring, you cannot directly see the features, so you

must interpret data indirectly, through probing procedures. The experimenters focused on the

amount and type of guidance needed within the highly spatial simulation.

63


Though a series of experiments, Mayer et al. (2002) found that pictorial scaffolding, as

opposed the verbal scaffolding, is needed to enhance performance in a visual-spatial task. In the

final experiments, participants were divided into verbal scaffolding, pictorial scaffolding, both,

and no scaffolding. Participants who received pictorial scaffolding solved significantly more

problems than those who did not receive pictorial scaffolding. Students who received strategic

scaffolding did not solve significantly more problems than students who did not receive strategic

scaffolding. While high-spatial participants performed significantly better than low-spatial

students, adding pictorial scaffolding to the learning materials helped both low- and high-spatial

students learn to use the Profile Game. Students in the pictorial-scaffolding group correctly

solved more transfer problems than students in the control group. However, pictorial scaffolding

did not significantly affect the solution time (speed) of either low- or high-spatial participants.

Overall, adding pictorial scaffolding to the learning materials lead to improved performance on a

transfer task for both high- and low-spatial students in the Profile Game (Mayer et al., 2002).

Contiguity effect

The contiguity effect addresses the cognitive load imposed when multiple sources of

information are separated (Mayer & Moreno, 2003; Mayer, Moreno, Boire, & Vagge, 1999;

Mayer & Sims, 1994; Moreno & Mayer, 1999). There are two forms of the contiguity effect:

spatial contiguity and temporal contiguity. Temporal contiguity occurs when one piece of

information is presented prior to other pieces of information (Mayer & Moreno, 2003; Mayer et

al., 1999; Moreno & Mayer, 1999). Spatial contiguity occurs when modalities are physically

separated (Mayer & Moreno, 2003). This study is concerned with spatial contiguity, since the

printed navigation maps will be spatially separated from the 3-D video game environment.

Contiguity results in split-attention (Moreno & Mayer, 1999).

64


Split Attention Effect

When dealing with two or more related sources of information (e.g., text and diagrams), it’s

often necessary to integrate mentally corresponding representations (verbal and pictorial) to

construct a relevant schema to achieve understanding. When different sources of information are

separated in space or time, this process of integration may place an unnecessary strain on limited

working memory resources (Atkinson et al., 2000; Mayer & Moreno, 1998). In this study, the

printed navigation maps are spatially separated from the 3-D video game environment, thereby

inducing the split-attention effect.

Summary of scaffolding

Depending upon the researcher, scaffolding has several meanings: the process of forming

and building upon a schema (Chalmers, 2003); all devices or strategies that support learning (van

Merrionboer et al., 2003), the process of diminishing support as learners acquire expertise (van

Merrionboer et al., 2002); and the process of training a student on core concepts and then

gradually expanding the training. What these four definitions have in common is that scaffolding

is related to providing support during learning.

Clark (2001) described instructional methods as external representations the external

processes of selecting, organizing, and integrating. Instructional methods also provide learning

goals (Alessi, 2000; Clark, 2001), monitoring (Clark, 2001), feedback (Alessi, 2000; Clark,

2001; Leemkuil et al., 2003), selection (Alessi, 2000; Clark, 2001), hints and prompts (Alessi,

2000), and various advance organizers (Jones et al., 1995). Each of these components either

reflects a form of scaffolding or reflects a need for scaffolding.

Mayer et al (2002) argued that a major instructional issue in learning by doing within

simulated environments concerns the proper type of guidance (i.e., scaffolding). One form of

65


scaffolding is graphical scaffolding. According to Allen (1997), selection of appropriate text and

graphics can aid the development of mental models, and Jones et al. (1997) commented that

visual cues such as maps help learners conceptualize the organization of the information in a

program (i.e., the learning space). A number of studies have supported the use of maps as visual

aids and organizers (Benbasat & Todd, 1993: Chou & Lin, 1998; Ruddle et al, 1999, Chou et al.,

2000; Farrell & Moore, 2000-2001)

Graphic organizers organize information in a graphic format, which act as spatial displays

of information that can also act as study aids (Chalmers, 2003), and Jones et al. (1995) argued

that interactive designers should provide users with visual or verbal cues to help them navigate

through unfamiliar territory. Cobb argued that cognitive load can be distributed to external media

(Cobb, 1997). In virtual environments, navigation maps help to navigate and to orient the user,

and facilities an easier learning experience (Yair et al, 2001). While navigation maps provide

external representations of information needed to complete tasks and to learn, thereby reducing

or distributing cognitive load (Cobb, 1997), they also have the potential to add load, ultimately

counteracting their positive effects. The contiguity effect addresses the cognitive load imposed

when multiple sources of information are separated (Mayer et al., 1999). Spatial contiguity

occurs when modalities are physically separated (Mayer & Moreno, 2003), such as a video game

screen and a printed navigation map. The split attention effect, which is related to the contiguity

effect, occurs when dealing with two or more related sources of information (e.g., information on

a screen and information on a printed navigation map). When different sources of information

are separated in space or time, this process of integration may place an unnecessary strain on

limited working memory resources (Atkinson et al., 2000). Therefore, when working with

66


navigation maps, careful consideration must be included for the potential adverse effects of

having the navigation map separated from the learning environment.

Summary of the Literature Review

Cognitive Load Theory is based on the assumptions of a limited working memory with

separate channels for auditory and visual/spatial stimuli, and a virtually unlimited capacity long-

term memory that stores schemas of varying complexity and level of automation (Brunken et al.,

2003). According to Paas et al. (2003), cognitive load refers to the amount of load placed on

working memory. Cognitive load can be reduced through effective use of the auditory and

visual/spatial channels, as well as schemas stored in long-term memory.

Meaningful learning is defined as deep understanding of the material and is reflected in

the ability to apply what was taught to new situations; i.e., problem solving transfer. (Mayer &

Moreno, 2003). Meaningful learning requires effective metacognitive skills: the management of

cognitive processes (Jones, Farquhar, & Surry, 1995), including selecting relevant information,

organizing connections among the pieces of information, and integrating (i.e., building) external

connections between incoming information and prior knowledge that exists in long-term memory

(Harp & Mayer, 1998). Related to meaningful learning is problem solving, “cognitive processing

directed at transforming a given situation into a desired situation when no obvious methods of

solution is available to the problem solver” (Baker & Mayer, 1999, p. 272). O’Neil’s Problem

Solving model (O’Neil, 1999) defines three core constructs of problem solving: content

understanding, problem solving strategies, and self-regulation. Each of these components is

further defined with subcomponents.

There are three types of cognitive load that can be defined in relationship to a learning or

problem solving task: intrinsic cognitive load (load from the actual mental processes involved in

67


creating schema), germane cognitive load (load from the instructional processes that deliver the

to-be-learned content), and extraneous cognitive load (all other load). An important goal of

instructional design is to balance intrinsic, germane, and extraneous cognitive loads to support

learning outcomes (Brunen et al., 2003).

Learner control, which is inherent in interactive computer-based media, allows for control

of pacing and sequencing (Barab, Young, & Wang, 1999). It also can induce cognitive overload

in the form of disorientation; loss of place (Chalmers, 2003), and is a potential source for

extraneous cognitive load. These issues may be the cause of mixed reviews of learner control

(Bernard, et al, 2003; Niemiec, Sikorski, & Wallberg, 1996; Steinberg, 1989), particularly in

relationship to novices versus experts (Clark, 2003c).

Computer-based educational games fall into three categories: games, simulations, and

simulation games. Games are linear, consist of rules, can contain imaginative contexts, are

primarily linear, and include goals as well as competition, either against other players or against

a computer (Gredler, 1996). Simulations display the dynamic relationship among variables

which change over time and reflect authentic causal processes and have a goal of discovering

causal relationships through manipulation of independent variables. Simulation games are a

blend of games and simulations (Gredler, 1996).

Clark (2003d) argued that mental effort is affected by motivation, and beginning with the

work of Malone (1981), a number of constructs have been described as providing the

motivational aspects of games: fantasy, control and manipulation, challenge and complexity,

curiosity, competition, feedback, and fun. Fantasy evokes “mental images of physical or social

situations that do not exist” (Malone & Lepper, 1987, p. 250). Control and manipulation promote

intrinsic motivation, because learners are given a sense of control over their choices and actions

68


(deCharms, 1986, Deci, 1975). Challenge embodies the idea that intrinsic motivation occurs

when there is a match between a task and the learner’s skills (Bandura, 1977, Csikszentmihalyi,

1975; Harter, 1978). For challenge to be effective, the the task should be neither too hard nor too

easy, otherwise the learner would lose interest (Clark, 1999; Malone & Lepper, 1987). Curiosity

is related to challenge and arises from situations in which there is complexity, incongruity, and

discrepancy (Davis & Wiedenbeck, 2001).

Studies on competition with games and simulations have resulted in mixed findings, due

to individual learner preferences, as well as the types of reward structures connected to the

competition (e.g., Porter, Bird, & Wunder, 1990-1991; Yu, 2001). Another motivational factor in

games, feedback, allows learners to quickly evaluate their progress and can take many forms,

such as textual, visual, and aural (Rieber, 1996). Ricci et al. (1996) argued that feedback can

produce significant differences in learner attitudes, resulting in increased attention to a learning

environment. However, Clark (2003a) commented that feedback must be focused on clear

performance goals and current performance.

The last category contributing to motivation, fun, is possibly an erroneous category. Little

empirical evidence exists for the construct. However, evidence does support the related

constructs of play, engagement, and flow. Play is entertainment without fear of present of future

consequences (Resnick & Sherer, 1994). Csikszentmihalyi (1975; 1990) defines flow as an

optimal experience in which a person is so involved in an activity that nothing else seems to

matter. According to Davis and Wiedenbeck (2001), engagement is the feeling of working

directly on the objects of interest in a world, and Garris et al. (2002) argued that engagement can

help to enhance learning and accomplish instructional objectives.

69


While numerous studies have cited the learning benefits of games and simulations (e.g.,

Adams, 1998; Baker et al., 1997; Betz, 1995-1996; Khoo & Koh, 1998), others have found

mixed, negative, or null outcomes from games and simulations, specifically in relationship to the

of enjoyment of a game to learning from the game (e.g., Brougere, 1999; Dekkers & Donatti,

1981; Druckman, 1995). There appears to be consensus among a large number of researcher with

regards to the negative, mixed, or null findings, suggesting that the cause might be a lack of

sound instructional design embedded in the games (de Jong & van Joolingen, 1998; Garris et al.,

2002; Gredler, 1996; Lee, 1999; Leemkuil et al., 2003; Thiagarajan, 1998; Wolfe, 1997). Among

the various instructional strategies, reflection and debriefing have been cited as critical to

learning with games and simulations.

An important component of research on the effectiveness of educational games and

simulations is the measurement and assessment of performance outcomes from the various

instructional strategies embedded into the games or simulations, such as problem solving tasks.

Problem solving is cognitive processing directed at achieving a goal when no solution method is

obvious to the problem solver (Baker & Mayer, 1999). O’Neil’s Problem Solving model (O’Neil,

1999), includes the components: content understanding; solving strategies—domain-independent

(-general) and domain-dependent (-specific); and self-regulation which is comprised of

metacognition and motivation. Metacognition is further composed of self-checking/monitoring

and planning, and motivation is comprised of effort and self-efficacy. Knowledge maps are

reliable and efficient for the measurement of the content understanding portion of O’Neil’s

Problem Solving model, and CRESST has developed a simulated Internet Web space to evaluate

problem solving strategies such as information searching strategies and feedback inquiring

strategies.

70


Problem-solving can play a large amount of cognitive load on working memory.

Instructional strategies have been recommended to help control or reduce that load. One such

strategy is scaffolding. While there are a number of definitions of scaffolding (e.g., Chalmers,

2003; van Merrionboer et al., 2002; van Merrionboer et al., 2003), what they all have in common

is that scaffolding is an instructional method that provides support during learning. Clark (2001)

described instructional methods as external representations the external processes of selecting,

organizing, and integrating. They provide learning goals, monitoring, feedback, selection, hints,

prompts, and various advance organizers (Alessi, 2000; Clark, 2001; Jones et al., 1995; Leemkuil

et al., 2003). Each of these components either reflects a form of scaffolding or reflects a need for

scaffolding

One form of scaffolding is graphical scaffolding. A number of studies have reported the

benefits of maps, which is a type of graphical scaffolding (Benbasat & Todd, 1993: Chou & Lin,

1998; Ruddle et al, 1999, Chou et al., 2000; Farrell & Moore, 2000-2001). In virtual

environments, navigation maps help to navigate and to orient the user, and facilities an easier

learning experience (Yair et al, 2001). While navigation maps can reduce or distribute cognitive

load (Cobb, 1997), they also have the potential to add load, ultimately counteracting their

positive effects. The spatial contiguity effect addresses the cognitive load imposed when multiple

sources of information are separated (Mayer & Moreno, 2003) and the split attention effect,

which is related to the contiguity effect, occurs when dealing with two or more related sources of

information (Atkinson et al., 2000). Navigation maps can provide value cognitive support for

navigating virtual environments, such as computer-based video games. This can be particularly

useful when using the gaming environment to accomplish a complex problem-solving task.

71


CHAPTER III: METHODOLOGY

Research Design

University of Southern California Human Subjects approval was requested on June 17,

2004. Revisions were requested on July 15 for the recruitment flyer and the Informed Consent

Form. Changes to these two forms were made and resubmitted on July 22, 2004. On July 26,

2005, the USC Institutional Review Board (IRB) approved all forms, allowing subjects to be

contacted and the experimental phase of study to begin.

The design of the study is true experimental, with randomized selection and assignment

of participants, and involves pre-, intermediate-, and post-tests for one treatment group and one

control group. Subjects were randomly assigned to either the treatment or the control group.

Group sessions involved only one group type: either all treatment participants or all control

participants. Due to limited availability of computers, session size was limited to a maximum of

three participants. At the end of the session, participants were debriefed and allowed to continue

playing on their own for up to 30 additional minutes (to examine continued motivation).

Research Questions and Hypotheses

Research Question 1: Will the problem solving performance of participants who use a

navigation map in a 3-D, occluded, computer-based video game (i.e., SafeCracker®) be better

than the problem solving performance of those who do not use the map (the control group)?





72







Research Question 2: Will the continued motivation of participants who use a

navigation map in a 3-D, occluded, computer-based video game (i.e., SafeCracker®) be greater

than the continued motivation of those who do not use the map (the control group)?



Sample

Two samples were selected solicited for this study. The first sample consisted of two

participants for the pilot study conducted September 28 and 29, 2004. The purpose of the study

was to review the procedures and instruments that were to be utilized in the main study. This

sample was a convenience sample and consisted of one female approximately 49 years 4 months

of age and a male approximately 32 years and 8 months of age. The female was a graduate of the

University of Southern California, Los Angeles, California. The male was a graduate of

California State University, Northridge, California. Both participants had a reasonable level of

computer proficiency and neither had prior experience with the game SafeCracker©.

Between September, 28, 2004 and March 21, 2005, sixty-six English-speaking adults,

ranging in age from 19 years and 4 months to 31 years and 11 months, participated in the main

study. The average participant age was a few days less than 23 years. All the participants for the

73


main study were undergraduate students, graduates, or graduate students of the University of

Southern California, Los Angeles. California.

Solicitation of participants. Their participation was solicited through several methods.

The primary method was a small standard paper size (8 and a half by 11 inch) flyer (Appendix

A) posted in various locations within five of the university’s schools; the Marshall School of

Business, the Viterbi School of Engineering, the Annenberg School for Communication, the

School of Cinema, and the Rossier School of Education. These schools were chosen for a

number of reasons, including; how many locations the allowed flyers to be posted at; ease of, or

ability to get, approval to post flyers; a belief that their students might be interested in

participating in a video game study.

Flyers (Appendix A) were also sent via email attachment to two of the university’s

student organizations that the researcher believed would contain students potentially interested in

this type of study. The organizations were a student video game development group and a

student television and film special effects group. The researcher is the faculty advisor to the

video game group. Flyers were also posted around campus at locations approved for display of

announcements. These locations included student congregation areas, major outdoor pathways,

and parking structure stairwells. The flyer included a statement that participants would be paid

$15 for approximately 90 minutes of participation, and participants must have no prior

experience playing the personal computer (PC)-based video game SafeCracker® (Daydream

Interactive, 1995). Email contact information was provided on the flyer.

Randomized assignment. As requests for participation arrived, each participant was

randomly assigned to either the treatment or control group. Participants were entered into a

Microsoft Excel 2002 spreadsheet for tracking purposes, in the order in which their emails were

74


received. Randomized assignment was accomplished using a random number generator within a

Microsoft Excel 2002 spreadsheet. The spreadsheet was used for tracking and other logistical

issues related to the study. When the participant’s last name was entered and either the enter or

tab key was pressed on the computer, The generator would display a number between

0.000000000 and 1.000000000, in increments of .000000001. If the number was from

0.000000000 to 0.500000000, the participant was assigned to the Control group. If the number

was from 0.510000000 and 1.000000000, the participant was assigned to the Treatment group.

Various study times were selected for each group and students were sent those times relevant to

their group. Participants responded by listing one or more times during which they could

participate. From the responses, the researcher scheduled participants to best fill each available

time slot.

Number of particpants whose data were analyzed. A total of 71 students participated in

the study, with 69 completing the study; Thirty-five received the treatment and thirty-four were

in the control group. Those in the treatment group received a navigation map, which was an

overhead view of the game’s playing environment; a mansion (figure 4). Those in the treatment

group also received instruction on how to read the map and use the map for planning and

navigation. Those in the control group did not receive the navigation map and were only given

brief instruction on how to navigate without a map.

Of the 71 students that participated in the study, 69 completed the study, but the data

from only 66 participants are included in this study. In one session involving two participants,

the researcher inadvertently had the participants overwrite a file with some of their prior data,

making the comparison between intermediate data and post data impossible. Those two

participants were not included in the data analysis. Three other participants did not complete the

75


study, due to computer errors. Near the end of the experimental phase of the study, two of the

computer began to exhibit problems. One computer began to freeze (quit accepting or responding

to user input). The other computer began to intermittently display an error during the second

round of game play during a session. In most cases, turning off the computers before various

phases of the study alleviated or prevented problems. However, near the end of the data

collection phase of the study, the computer that had intermittently been freezing began to

regularly freeze. Two participants had to end the study early and not enough data was collected

both either participant to be analyzable. From that point onward, that computer was not used in

the study, limiting the number of participants for any session to two. For the other computer,

restarting the computer at various phases worked well for all but one participant. This participant

had to leave the study early and not enough data was collected for analysis. It should also be

noted that two participants had to leave very early in the study due to computer problems.

However, in both cases, the subjects had only been shown how to use one software package (the

Knowledge Mapping software). It was determined to be acceptable to have those two subjects

return to complete the study at a later date without compromising the validity of their data. They

did return and complete the study.

Hardware

The pilot study took place in the home office of the researcher. The computer utilized for

the study was a 450 MHz (megahertz) desktop computer made by Tiger Direct

(http://www.tigerdirect.com) with 64 MB (megabytes) of RAM (random-access memory), a

standard computer keyboard, a 3-button mouse, and a 21” CRT (cathode-ray tube) monitor.

The main study took place in the campus office of the researcher. A table was set up for

two computers placed side by side. One computer was a Pentium 200, NeTPower Symmetra

76


computer with 128 MB of RAM that originally ran the Windows NT operating system, but was

installed with the Windows 98 operating system for the study. The computer configuration

included a standard computer keyboard, a 3-button mouse, and a 17” CRT monitor. Another

computer was a Sony PCG-F520, Pentium III laptop computer with 192MB RAM, running the

Windows 98 operating system. The laptop’s keyboard was used, but an external USB (universal

serial-bus) 2-button mouse was added. A 14” CRT monitor was attached to this computer,

because the laptop’s built-in LCD (liquid crystal display) monitor was not very good; the

displayed images were very dark and had low contrast, and the screen exhibited a lot of

reflection and glare, making visibility difficult. This computer was placed on the table next to the

NeTPower Symmetra. The Symmetra’s power case was placed on the desk between the two

computers, to reduce the visibility by participants of each other’s monitor. The third computer

was a Dell Latitude D500, Pentium M laptop computer with 256MB RAM, and running the

Windows 98 operating system. Similar to the other laptop, this laptop’s keyboard was used, but a

serial bus 2-button mouse was added. This computer was placed on a lateral file cabinet. A

monitor was not attached to this laptop, because its built-in 12” LCD screen produced a

satisfactory picture. The three computers were placed so that participants could not readily see

what other participants were doing and participants had enough room to use the mouse and to

write on paper. The primary mode of computer input and interaction during the study was via the

mouse. The only time the keyboard was used was for entering a file name when saving various

types of data. Two files were saved during one phase of the study and two files were saved

during a second phase of the study, for a total of four files.

Instruments

77


A number of instruments will be included in the study: a demographic/game preference

form, a task completion form, a self-regulation questionnaire, SafeCracker®, a navigation map

of the game’s environment, problem-solving retention and transfer questionnaire; and

knowledge mapping software.

Demographic, game play, and game preference questionnaire

At the start of the experiment, this questionnaire (Appendix B) was administered to elicit

gender, age, amount of weekly video game play, and preferred game types. For gender,

participants marked either the male or female box. For age, participants entered both the number

of years and the number of months. For amount of weekly video game play, participants checked

one of four boxes: none, 1 to 2 hours, 3 to 6 hours, and greater than 6 hours. The game types

section listed Puzzle games, RTS games, FPS games, Strategy games, Role Playing games,

Arcade games, PC games, and Console games. The first five items in this section are game

genres. The last two items are game platforms. The sixth item, Arcade game, can be either a

game genre or platform. For each of the preferred game types, participants entered a number

from 1 to 5, with 1 indicating low interest and 5 indicating high interest. Participants were

prompted to enter a zero if they did not play that game type, were not sure of their response, or

did not know the particular type of game type or what the initials meant. It was determined by

the researcher that those who played RTS or FPS games, in particular, would know what those

terms meant, since the terms were commonly used by players of those particular game types;

RTS stands for Real-Time Strategy and FPS stands for First-Person Shooter (also known as first

person perspective). If a participant asked what a term meant, he or she was told to enter a zero.

The last two game types are gaming platforms. PC games, which stand for Personal

Computer games, refer to games played on personal, or home, computers (PCs), such as an

78


Apple computer or Windows-based computer. Console games are those games played on gaming

consoles, such as PlayStation, X-Box, or Nintendo game consoles.

Arcade games refers to both a genre and a platform. As a genre, arcade games are short,

and often rapid reaction, games with short playing durations, and only one or two goals. As a

game platform, arcade games historically refers to games played on large stand-alone gaming

consoles like those found in public arcades. Today, however, arcade games are also available on

home computers (PCs).

The divisions for amount of weekly game play included in the questionnaire (none, 1 to 2

hours, 3 to 6 hours, and greater than 6 hours) were based on a study conducted in 1996 by the

Media Analysis Laboratory, Simon Fraser University, Burnaby, British Columbia, Canada. The

study surveyed 647 children ranging in age from 11 to 18, with 80% between the ages of 13 and

15. Six hundred forty six subjects completed the survey (351 male and 295 female). Most

children surveyed in this study played video games between 1 and 6 hours per week. Based on

the findings of this study, which indicated that most children surveyed played between 1 and 6

hours per week, the four divisions used in this study were created. Information on the British

Columbia study can be found at http://www.mediaawareness.ca/english/resources/research_

documents/studies/video_games/vgc_vg_and_television.cfm

Task completion form

Immediately before the start of each game (the game was played twice during the study),

participants were handed a Task Completion form (Appendix C). This form listed the names of

the rooms that were involved in a particular game and the safes that could be found in each

room. Players were told to mark off (check the boxes for) each safe they opened and to be sure to

mark them off as soon as a safe was opened, so as not to forget which safes were opened during

79


a game. At the end of each game, players were prompted to check the form to ensure all opened

safes were marked off.

Self-Regulation Questionnaire

A trait self-regulation questionnaire (Appendix D) designed by O’Neil and Herl (1998)

was administered to assess each participant’s degree of self-regulation, which is one of the three

components of problem-solving ability as defined by O’Neil (1999). Reliability of the instrument

ranges from .89 to .94, as reported by O’Neil & Herl (1998). The 32 items on the questionnaire

are composed of eight items each of the four self-regulation factors in the O’Neil (1999) Problem

Solving model (see figure 1): planning, self-checking/monitoring, self-efficacy, and effort. For

example, item 1 (Appendix D) “I determine how to solve a task before I begin.” is designed to

assess a participant’s planning ability; and item 2 “I check how well I am doing when I solve a

task” is to evaluate a participant’s self-efficacy. The response format for each item is a Likert-

type scale with four possible responses; almost never, sometimes, often, and almost always.

The self-regulation form was administered in printed format, with participants using

either a pen or pencil to enter responses (a number from 1 to 4) for each question. Responses

were later entered into a Microsoft Excel 2002 spreadsheet and totals for each for the four self-

regulation factors were generated using Excel’s SUM function.

SafeCracker

The non-violent, PC-based video game SafeCracker® (Daydream Interactive, 1995) was

selected for this study, as a result of a feasibility study by Wainess and O’Neil (2003). The

purpose of the feasibility study was to recommend a video game for use as a platform for

research on cognitive and affective components of problem solving, based on the O’Neil (1999)

80


Problem Solving Model (see figure 1). The following requirements were established for a game

to be considered as a potential platform.

Required characteristics:

* Adult focused (college age and above)

* Appeal to both genders and with broad audience appeal

* Single user play

* Suitable for problem solving research

* Game to support practice and include problem solving feedback

* Interface, controls, and navigation can be adequately mastered in half an hour or less

* Can be played multiple times (or multiple stages) in one hour

* Compatible with X-Box, Playstation 2, or PC

Expert support available for analysis of play/problem solving

The following additional criteria are preferred, but not required.

* Non-violent game

* User controlled pacing of game action

* Multi-user (multiplayer) capable

* Suitable for study of retention and transfer

* Provides multiple conditions or can be programmed with varying conditions

* Ability to track actions (e.g. time, clicks, choices)

The feasibility study by Wainess and O’Neil (2003) involved four phases: 1) selection of

relevant game types, based on simple search criteria; 2) in-depth analysis of the most relevant

game selections; 3) removal of similar games by selecting the most appropriate game among the

equivalent games; 4) play testing of each remaining game to determine the final selection. Phase

81


1 began with selection of 525 potential games and ended with the selection of 12 possible games.

Phase 2 ended with the selection of four possible games, and phase 3 ended with the selection of

three possible games: SafeCracker for Windows®, Myst III®: Exile for Windows/PS2/Xbox©,

and Jewels of the Oracle for Windows©. Phase 4 resulted in the selection of SafeCracker®

(Wainess & O’Neil, 2003).

According to Wainess and O’Neil (2003), the primary factor for selecting SafeCracker®

over the other two possible games selected during phase 3 was time constraints. During the

feasibility study, it had been decided that the participants should not be required to spend more

than one and a half hours on the study. In addition, it was desirable to include multiple iterations

of gameplay within that time period. In Phase 4, Myst III®: Exile© and Jewels of the Oracle©

were both eliminated due to the number of hours required for problem solving research. The

games are designed with large environments, numerous buildings and other structures, and

myriad, complex clues that would require multiple hours of play time to accomplish a problem

solving task. With SafeCracker, players would be able to learn the controls and interface, and

enter the main environment (a mansion), within a matter of minutes. Because only two or three

of the mansion’s approximately 50 rooms are needed for an effective study, problem solving

events could occur in 10 or 20 minutes, allowing for multiple problem-solving tasks using

different room combinations.

Navigation map

Gameplay in SafeCracker takes place in a two story mansion. For the purposes of this

study, only a small number of rooms on the first floor were utilized. A navigation map, in the

form of a topological floor plan of the first floor, was downloaded from

http://www.gameboomers.com/wtcheats/pcSs/Safecracker.htm. The navigation map was

82


subsequently modified using Adobe® Photoshop® 6.5, to alter the view of the navigation map

from one-point perspective to a flat 2-D image, to remove unnecessary artifacts, to remove room

numbers for each room, add a name to each room in accordance with names displayed on the

game’s interface, and add a compass symbol to the top right side of the map. For the final,

modified version of the map, see figure 4, below.

Figure 4: Navigation Map

Knowledge Map

In this research, participants were asked to create a knowledge map in a computer-based

environment to evaluate their content understanding after playing SafeCracker. During the study,

participants played SafeCracker twice and completed a knowledge map after each game session.

The computerized knowledge map used in this study had been successfully applied to other

studies (e.g., Chuang, 2003; Chung et al., 1999; Hsieh, 2001; Schacter et al., 1999). Appendix B

83


lists the knowledge map specification that will be used in this study (adapted from Chen, in

preparation)

Content understanding measure.

Content understanding measures were computed by comparing the semantic content

score of a participant’s knowledge map to the average semantic score of two experts. The experts

for this study are Richard Wainess and Hsin-Hu (Claire) Chen. Appendix C shows the concept

map developed for this study. It is based on the general concepts and propositions relevant to

problem solving in the game, not the concepts and propositions specific to a room or a safe. An

example of the knowledge map for one room in this study is shown in Figure 5.

Figure 5: Knowledge map for one SafeCracker® room

The following describes how these outcomes will be scored. First, the semantic score is

calculated based on the semantic propositions—two concepts connected by one link, in experts’

knowledge map. Every proposition in a participant’s knowledge map will be compared against

each proposition in the experts’ map. A match would be scored as one point. The average score

of the two experts would be the semantic score for the student map. For example, as seen in

containsroom key

crack

cluemap

results inresults from

causes

84


Table 1, if a participant made a proposition such as “room”, this proposition would then

compared against the experts’ propositions.

Table 1: An Example of Scoring Map

CONCEPT1 LINKS CONCEPT 2 EXPERT 1 EXPERT 2 AVERAGE

Room contains key 1 1 1

Crack results

from

Key 1 0 0.5

Clue causes Crack 0 1 0.5

Total 2.00

The participant can only receive one of two scores: the average score of the two experts

or a score of zero. In the case of “room,” a score of one (the average score of the two experts)

would mean the participant’s proposition was the same as the proposition of at least one expert.

A score of zero would mean the proposition does not match either expert’s proposition. If the

participant then created the link “room contains key” he or she would receive an addition score

of one point to reflect the average score of the two experts, for a total of two points (one point for

the semantic link “room” and one point for the proposition “room contains key”). Selecting

“crack” would receive one point (for matching at least one expert). Creating the link (the

proposition) “crack results in key” would garner an additional half-point (the average score of

the two experts), for a total of one and one-half points.

Domain-Specific Problem-Solving Strategies Measure

In this study, the researcher modified Mayer and Moreno’s (1998) problem-solving

question list to measure domain specific problem-solving strategies. In Mayer and Moreno’s

(1998) research on the split-attention effect in multimedia learning and the dual processing

85


systems in working memory, participants’ problem-solving strategies were assessed with a set of

retention and transfer questions. Mayer and Moreno (1998) judged a participant’s retention

score by counting the number of predefined major idea units correctly stated by the participant,

regardless of wording. Examples of the predefined answer units for retention were “air rises”,

“water condenses”, “water and crystals fall”, and “wind is dragged downward” (p. 315).

Similarly, Mayer and Moreno (1998) scored the transfer questions by counting the

number of acceptable answers that the participant produced across all of the transfer problems.

For example, an acceptable answer for the first transfer question about decreasing lightning

intensity was “removing positive ions from the ground”, and one of the acceptable answers for a

question about the reason for the presence of clouds without lightning was that “the tops of the

clouds might not be high enough to freeze” (p. 315).

The problem-solving strategies questions designed for this dissertation research have

been adapted for relevance to the tasks associated with Safecracker: finding and opening safes.

Furthermore, the questions are related to the application of strategies relevant to the puzzles and

safes, and solving or cracking strategies participants may acquire after trying to solve the

problems in the rooms they were assigned. Figure 6 shows the problem solving strategy

questions for retention and transfer which were used in this dissertation research (modified from

the questions developed by Chen, in preparation).

Figure 6: Retention and Transfer Questions

Retention question: List the ways you found rooms and opened safes.

Transfer question: List some ways to improve the design of the game play for opening

safes.

86


Each question was presented in printed form on standard 8 and a half inch by 11 inch

paper. Following each question were numbered lines for responses. Participants were told to

write shot responses. They were told to begin with the retention question and not to go to the

response question until told to do so. Unlike in the Mayer and Moreno (1998) study, instead of

giving each response one point and totaling the score for each question, responses were

weighted, based on the taxonomy table (figure 7) found in Anderson et al (2001). The table lists

six cognitive process dimensions in columns with more meaningful dimensions to the left and

less meaningful dimensions to the left. Meaningfulness referred to the depth of learning that has

occurred or that is required in order to provide a response that meets the requirements for a

column. For example, being able to apply knowledge (column number 3) indicates more

understanding of a subject matter than simply remembering facts (column number 1).

The knowledge dimensions, listed in four rows, refer to varying levels (or depths) of

knowledge, with the top row (row A) indicating the lowest depth of knowledge and the lowest

row (row D) indicating the greatest depth of knowledge. For example, factual knowledge (row

A) refers to recognizing something, while procedural knowledge (row C) refers to knowing the

steps involved in a process or procedure. Procedural knowledge requires a greater amount of

knowledge than does factual knowledge.

Figure 7: Taxonomy Table

The

Knowledge

Dimension

The cognitive process dimension

1.

Remember

2.

Understand

3.

Apply

4.

Analyze

5.

Evaluate

6.

Create

A.

Factual

87


Knowledge

B.

Conceptual

Knowledge

C.

Procedural

Knowledge

D.

Meta-

Cognitive

Knowledge

Because each column, from left to right, indicates a greater depth of knowledge or

understanding, and because each row, moving downward, also indicates a greater depth of

knowledge or understanding, a point value was given to each box defined by the intersection of a

row and a column with values increasing from left to right and from top to bottom. Figure 8

shows the Taxomony of Learning Table with knowledge values in each box. In row A, the values

range from 1 on the left to 6 on the right. In row B, the values range from 2 on the left to 7 on the

right. In row C, the values range from 3 on the left to 8 on the right. In row D, the values range

from 4 on the left to 9 on the right. With this pattern, boxes in several rows contain the same

values. Through review of Anderson et al (2001), two researchers, the researcher from this study

and the research from a related study Shen (in preparation) determined that the knowledge levels

exhibited in various tables could be considered equivalent (or at least similar). For example, row

88


A, column 2 and row B, column 1 both have a value of 2. According to Wainess and Shen, the

depth of knowledge needed to display understanding of factual knowledge (row A, column 2) is

similar to the amount of knowledge needed to remember concepts (row B, column 1). In another

example, the depth of knowledge needed to evaluate facts (row A, column 5) is similar to or

equal to the amount of knowledge necessary to analyze concepts (row B, column 4), apply

procedural knowledge (row C, column 3), or understand ones own mental processes (row D,

column 2).

Figure 8: Taxonomy Table with knowledge values

The

Knowledge

Dimension

The cognitive process dimension

1.

Remember

2.

Understand

3.

Apply

4.

Analyze

5.

Evaluate

6.

Create

A.

Factual

Knowledge

1 2 3 4 5 6

B.

Conceptual

Knowledge

2 3 4 5 6 7

C.

Procedural

Knowledge

3 4 5 6 7 8

D.

Meta-

Cognitive

4 5 6 7 8 9

89


Knowledge

Participants’ retention scores will be counted by the number of predefined major idea

units correctly stated by the participant regardless of wording. The example of the answer units

for the retention were “follow map”, “find clues”, “find key”, “differentiate rooms” and “tools

are cumulative”. Participants’ transfer questions will be scored by counting the number of

acceptable answers that the participant produced across all transfer problems. For example, the

acceptable answers for the first transfer question in figure 5 include “jot down notes”, and one of

the acceptable answers for question three for non-navigation map users would be “use the

compass.”

Procedure for the Study

The study consisted of introducing the participants to the objective of the experiment,

describing the experiment as an examination of performance in a video game environment, but

not discussing the issue of navigation maps. Following the introductions, participants will fill out

the demographic and self-regulation questionnaires, after which they will be introduced to the

knowledge mapping software and guided through its use.

Next the participants will be introduced to the game. They will be guided through starting

the game, use of the interface, navigation to the mansion, opening the lock on the gate outside

the mansion, opening the mansion’s front door, and entering the first room of the mansion. The

first room of the mansion contains two safes. One of the safes will already be open. Participants

will be guided though searching the room and acquiring at least one item.

90


Next the participants will fill out a knowledge map related to the game (the pre-test).

Following that, at a prescribed time, participants will begin playing the game; continuing the

search of the first room and finding the second room, where they will collect additional items

any safes that are found. They will again use the knowledge map (intermediate-test) and again

fill out the problem-solving strategy questionnaire (intermediate-test). In addition, they will

check off the task list for all tasks completed.

Once again, the participants will be given the task of locating two or three rooms and

opening any safes that are found. And once again, they will use the knowledge map (post-test),

fill out the problem-solving questionnaire (post-test), and check off the task list. A final step will

be to debrief the participants, during which participants will be asked if they would like to

continue playing on their own. If they say yes, they will be given up to an additional half-hour to

play. The purpose of this phase is to examine the motivational effects of the game, to determine

the relationship between navigation maps and continued motivation.

Timing Chart

Table 2 lists the activities encompassing the experiment and the times allocated with each

activity, and ending with total time.

Table 2: Time Chart of the Study

ACTIVITY TIME ALLOCATION

IntroductionAnd study related paperwork

5 minutes

Self-regulation questionnaire and demographic data 10 minutes

Introduction on knowledge mapping 10 minutes

Game introduction 10 minutes

Game playing (first 3 rooms) 15 minutes

91


And Task Completion Form

Knowledge map (intermediate) 7 minutes

Problem-solving strategy questions (intermediate) 4 minutes

Game playing (second 3 rooms)And Task Completion Form

15 minutes

Knowledge map (post) 7 minutes

Problem-solving strategy questions (post) 4 minutes

Debriefing 3 minutes

TOTAL 90 minutes

Additional play time Up to 30 minutes

92


CHAPTER IV: ANALYSIS AND RESULTS

Descriptive statistics (.e.g., means and standard deviations) will be used throughout the

study to describe all measures. The relationship of self-regulation and of presence or absence of a

navigation map to task difficulty, as measured by task completion, will be analyzed using

correlation analyses.



A t-test will be used to compare the effect of a navigation map on content understanding

as compared to no map.



A t-test will be used to compare the effect of a navigation map on problem strategy

retention as compared to no map.



A t-test will be used to compare the effect of a navigation map on problem strategy

transfer as compared to no map.




Pearson’s correlation will be used to assess the effect of the four self-regulation variables

(planning, self-checking/monitoring, self-efficacy, and effort) on the navigation map group as

compared to the control group.

93




A t-test will be used to compare the effect of a navigation map on continued motivation

as compared to no map.

94


REFERENCES

Adams, P. C. (1998, March/April). Teaching and learning with SimCity 2000 [Electronic

Version]. Journal of Geography, 97(2), 47-55.

Alessi, S. M. (2000). Simulation design for training and assessment. In H. F. O’Neil, Jr. & D. H.

Andrews (Eds.), Aircrew training and assessment (pp. 197-222). Mahwah, NJ: Lawrence

Erlbaum Associates.

Alexander, P. A. (1992). Domain knowledge: Evolving themes and emerging concerns.

Educational Psychologist, 27(1), 33-51.

Allen, R. B. (1997). Mental models and user models. In M. Helander, T. K. Landauer & P.

Prabhu (eds.), Handbook of Human Computer Interaction: Second, Completely Revised

Edition (pp. 49-63). Amsterdam: Elsevier

Anderson, R. C., & Pickett, J. W. (1978). Recall of previously recallable information following a

shift in perspective. Journal of Verbal Learning and Verbal Behavior, 17, 1-12.

Asakawa, T., Gilbert, N. (2003). Synthesizing experiences: Lessons to be learned from Internet-

mediated simulation games. Simulation & Gaming, 34(1), 10-22.

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples:

Instructional principles from the worked examples research. Review of Educational

Research, 70(2), 181-214.

Atkinson, R. K., Renkl, A., Merrill, M. M. (2003). Transitioning from studying examples to

solving problems: Effects of self-explanation prompts and fading worked-out steps.

Journal of Educational Psychology, 95(4), 774-783.

Ausubel, D. P. (1963). The psychology of meaningful verbal learning. New York: Grune and

Stratton.

95


Baddeley, A. D. (1986). Working memory. Oxford, England: Oxford University Press.

Baddeley, A. D., & Logie, R. H. (1999). Working memory: The multiple-component model. In

A. Miyake & P. Shah (Eds). Models of working memory: Mechanisms of active

maintenance and executive control (pp. 28-61). Cambridge, England: Cambridge

University Press.

Baker, D., Prince, C., Shrestha, L., Oser, R., & Salas, E. (1993). Aviation computer games for

crew resource management training. The International Journal of Aviation Psychology,

3(2), 143-156.

Baker, E. L., & Mayer, R. E. (1999). Computer-based assessment of problem solving. Computers

in Human Behavior, 15, 269-282.

Baker, E. L. & O’Neil, H. F., Jr. (2002). Measuring problem solving in computer environments:

current and future states. Computers in Human Behavior, 18, 609-622.

Banbury, S. P., Macken, W. J., Tremblay, S., & Jones, D. M. (2001, Spring). Auditory

distraction and short-term memory: Phenomena and practical implications. Human

Factors, 43(1), 12-29.

Barab, S. A., Bowdish, B. E., & Lawless, K. A. (1997). Hypermedia navigation: Profiles of

hypermedia users. Educational Technology Research & Development, 45(3), 23-41.

Barab, S. A., Young, M. F., & Wang, J. (1999). The effects of navigational and generative

activities in hypertext learning on problem solving and comprehension. International

Journal of Instructional Media, 26(3), 283-309.

Baylor, A. L. (2001). Perceived disorientation and incidental learning in a web-based

environment: Internal and external factors. Journal of Educational Multimedia and

Hypermedia, 10(3), 227-251.

96


Benbasat, I., & Todd, P. (1993). An experimental investigation of interface design alternatives:

Icon vs. text and direct manipulation vs. menus. International Journal of Man-Machine

Studies, 38, 369-402.

Berson, M. J. (1996, Summer). Effectiveness of computer technology in the social studies: A

review of the literature. Journal of Research on Computing in Education, 28(4), 486-499.

Berylne, D. E. (1960). Conflict, arousal, and curiosity. New York: McGraw-Hill.

Betz, J. A. (1995/1996). Computer games: Increase learning in an interactive multidisciplinary

environment. Journal of Educational Technology Systems, 24(2), 195-205.

Bong, M. (2001). Between- and within-domain relationships of academic motivation among

middle and high school students: Self-efficacy, task-value, and achievement goals.

Journal of Educational Psychology, 93(1), 23-34.

Brougere, G. (1999, June). Some elements relating to children’s play and adult

simulation/gaming. Simulation & Gaming, 30(2), 134-146.

Brown, D. W., & Schneider, S. D. (1992), Young learners’ reactions to problem solving

contrasted by distinctly divergent computer interfaces. Journal of Computing in

Childhood Education, 3(3/4), 335-347.

Brozik, D., & Zapalska, A. (2002, June). The PORTFOLIO GAME: Decision making in a

dynamic environment. Simulation & Gaming, 33(2), 242-255.

Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in

multimedia learning. Educational Psychologist 38(1), 53-61.

Brunning, R. H., Schraw, G. J., & Ronning, R R. (1999). Cognitive psychology and instruction

(3rd ed.). Upper Saddle River, NJ: Merrill.

97


Carroll, W. M. (1994). Using worked examples as an instructional support in the algebra

classroom. Journal of Educational Psychology, 86(3), 360-367.

Chalmers, P. A. (2003). The role of cognitive theory in human-computer interface. Computers in

Human Behavior, 19, 593-607.

Chen, H. H. (in preparation) A formative evaluation of the training effectiveness of a computer

game. Dissertation Proposal. University of Southern California.

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations:

How students study and use examples in learning to solve problems. Cognitive Science,

13, 145-182.

Chou, C., & Lin, H. (1998). The effect of navigation map types and cognitive styles on learners’

performance in a computer-networked hypertext learning system [Electronic Version].

Journal of Educational Multimedia and Hypermedia, 7(2/3), 151-176.

Chou, C., Lin, H, & Sun, C.-t. (2000). Navigation maps in hierarchical-structured hypertext

courseware [Electronic Version]. International Journal of Instructional Media, 27(2),

165-182.

Chuang, S.-h. (2003). The role of search strategies and feedback on a computer-based problem

solving task. Unpublished doctoral dissertation. University of Southern California.

Clark, R. E. (1999). The CANE model of motivation to learn and to work: A two-stage process

of goal commitment and effort [Electronic Version]. In J. Lowyck (Ed.), Trends in

Corporate Training. Leuven, Belgium: University of Leuven Press.

Clark, R. E. (2003a, February). Strategies based on effective feedback during learning. In H. F.

O’Neil (Ed.), What Works in Distance Education. Report to the Office of Naval Research

98


by the National Center for Research on Evaluation, Standards, and Student Testing (pp.

18-19).

Clark, R. E. (2003b, February). Strategies based on increasing student motivation: Encouraging

active engagement and persistence. In H. F. O’Neil (Ed.), What Works in Distance

Education. Report to the Office of Naval Research by the National Center for Research

on Evaluation, Standards, and Student Testing (pp. 20-21).

Clark, R. E. (2003b, February). Strategies based on providing learner control of instructional

navigation. In H. F. O’Neil (Ed.), What Works in Distance Education. Report to the

Office of Naval Research by the National Center for Research on Evaluation, Standards,

and Student Testing (pp. 14-15).

Clark, R. E. (2003d, March). Fostering the work motivation of teams and individuals.

Performance Improvement, 42(3), 21-29.

Clark, R. E. (Ed.).(2001). Learning from Media: Arguments, analysis, and evidence. Greenwich,

CT: Information Age Publishing.

Cobb, T. (1997). Cognitive efficiency: Toward a revised theory of media. Educational

Technology Research and Development, 45(4), 21-35.

Coffin, R. J., & MacIntyre, P. D. (1999). Motivational influences on computer-related affective

states. Computers in Human Behavior, 15, 549-569.

Corno, L., & Mandinach, E. B. (1983). The role of cognitive engagement in classroom learning

and motivation. Educational Psychologist, 18(2), 88-108.

Crookall, D., & Aria, K. (Eds.). (1995). Simulation and gaming across disciplines and cultures:

ISAGA at a watershed. Thousand Oaks, CA: Sage.

99


Crookall, D., Oxford, R. L., & Saunders, D. (1987). Towards a reconceptualization of

simulation. From representation to reality. Simulation/Games for Learning, 17, 147-171.

Cross, T. L. (1993, Fall). AgVenture: A farming strategy computer game. Journal of Natural

Resources and Life Sciences Education, 22, 103-107.

Csikszentmihalyi, M. (1975). Beyond boredom and anxiety. San Francisco: Jossey Bass.

Csikszentmihalyi, M. (1990). Flow: The psychology of optimal performance. New York:

Cambridge University Press.

Cutmore, T. R. H., Hine, T. J., Maberly, K. J., Langford, N. M., & Hawgood, G. (2000).

Cognitive and gender factors influencing navigation in a virtual environment.

International Journal of Human-Computer Studies, 53, 223-249.

Daniels, H. L., & Moore, D. M. (2000). Interaction of cognitive style and learner control in a

hypermedia environment. International Journal of Instructional Media, 27(4), 369-383.

Davis, S., & Wiedenbeck, S. (2001). The mediating effects of intrinsic motivation, ease of use

and usefulness perceptions on performance in first-time and subsequent computer users.

Interacting with Computers, 13, 549-580.

Day, E. A., Arthur, W., Jr., & Gettman, D. (2001). Knowledge structures and the acquisition of a

complex skill. Journal of Applied Psychology, 86(5), 1022-1033.

de Jong, T., de Hoog, R., & de Vries, F. (1993). Coping with complex environments: The effects

of providing overviews and a transparent interface on learning with a computer

simulation. International Journal of Man-Machine Studies, 39, 621-639.

de Jong, T., & van Joolingen, W. R. (1998). Scientific discovery learning with computer

simulations of conceptual domains. Review of Educational Research, 68, 179-202.

deCharms, R. (1986). Personal Causation. New York: Academic Press.

100


Deci, E. L. (1975). Intrinsic Motivation. New York: Plenum Press.

Dekkers, J., & Donati, S. (1981). The interpretation of research studies on the use of simulation

as an instructional strategy. Journal of Educational Research, 74(6), 64-79.

Dempsey, J. V., Haynes, L. L., Lucassen, B. A., & Casey, M. S. (2002). Forty simple computer

games and what they could mean to educators. Simulation & Gaming, 43(2), 157-168.

Dias, P., Gomes, M. J., & Correia, A. P. (1999). Disorientation in hypermedia environments:

Mechanisms to support navigation. Journal of Educational Computing Research, 20(2),

93-117.

Dillon, A., & Gabbard, R. (1998, Fall). Hypermedia as an educational technology: A review of

the quantitative research literature on learner comprehension, control, and style. Review

of Educational Research, 63(3), 322-349.

Donchin, E. (1989). The learning strategies project. Acta Psychologica, 71, 1-15.

Druckman, D. (1995). The educational effectiveness of interactive games. In D. Crookall & K.

Aria (Eds.), Simulation and gaming across disciplines and cultures: ISAGA at a

watershed (pp. 178-187). Thousand Oaks, CA: Sage

Eberts, R. E., & Bittianda, K. P. (1993). Preferred mental models for direct-manipulation and

command-based interfaces. International Journal of Man-Machine Studies, 38, 769-785.

Eccles, J. S., & Wigfeld, A. (2002). Motivational beliefs, values, and goals. Annual Review of

Psychology, 53, 109-132.

Frohlich, D. M. (1997). Direct manipulation and other lessons. In M. Helander, T. K. Landauer

& P. Prabhu (eds.), Handbook of Human Computer Interaction: Second, Completely

Revised Edition (pp. 463-488). Amsterdam: Elsevier

101


Galimberti, C., Ignazi, S., Vercesi, P., & Riva, G. (2001). Communication and cooperation in

networked environment: An experimental analysis. CyberPsychology & Behavior, 4(1),

131-146.

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and

practice model. Simulation & Gaming, 33(4), 441-467.

Gevins, A., Smith, M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R., & Rush, G. (1998).

Monitoring working memory load during computer-based tasks with EEG pattern

recognition methods. Human Factors, 40(1), 79-91.

Gredler, M.E. (1996). Educational games and simulations: a technology in search of a research

paradigm. In D. H. Jonassen (Ed.). Handbook of Research for Educational

Communications and Technology. (pp 521-540). New York: Simon & Schuster

Macmillan.

Green, C. S., & Bavelier, D. (2003, May 29). Action video game modifies visual selective

attention. Nature, 423, 534-537.

Greenfield, P. M., deWinstanley, P., Kilpatrick, H., & Kaye, D. (1996). Action video games and

informal education: Effects on strategies for dividing visual attention. In P. M. Greenfield

& R. R. Cocking (Eds.), Interacting with Video (pp. 187-205). Norwood, NJ: Ablex

Publishing Corporation.

Hannifin, R. D., & Sullivan, H. J. (1996). Preferences and learner control over amount of

instruction. Journal of Educational Psychology, 88, 162-173.

Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of

cognitive interest in science learning. Journal of Educational Psychology, 90(3), 414-

434.

102


Harter, S. (1978). Effectance motivation reconsidered: Toward a developmental model. Human

Development, 1, 34-64.

Henderson, L., Klemes, J., & Eshet, Y. (2000). Just playing a game? Educational simulation

software and cognitive outcomes. Journal of Educational Computing Research, 22(1),

105-129.

Herl, H. E., Baker, E. L., & Niemi, D. (1996). Construct validation of an approach to modeling

cognitive structure of U.S. history knowledge. Journal of Educational Psychology, 89(4),

206-218.

Herl, H. E., O’Neil, H. F., Jr., Chung, G., & Schacter, J. (1999) Reliability and validity of a

computer-based knowledge mapping system to measure content understanding. Computer

in Human Behavior, 15, 315-333.

Hong, E., & O’Neil, H. F. Jr. (2001). Construct validation of a trait self-regulation model.

International Journal of Psychology, 36(3), 186-194.

Howland, J., Laffey, J., & Espinosa, L. M. (1997). A computing experience to motivate children

to complex performances [Electronic Version]. Journal of Computing in Childhood

Education, 8(4), 291-311.

Hubbard, P. (1991, June). Evaluating computer games for language learning. Simulation &

Gaming, 22(2), 220-223.

Jones, M. G., Farquhar, J. D., & Surry, D. W. (1995, July/August). Using metacognitive theories

to design user interfaces for computer-based learning. Educational Technology, 35(4),

12-22.

103


Kaber, D. B., Riley, J. M., & Tan, K.-W. (2002). Improved usability of aviation automation

through direct manipulation and graphical user interface design. The International

Journal of Aviation Psychology, 12(2), 153-178.

Kagan, J. (1972). Motives and development. Journal of Personality and Social Psychology, 22,

51-66.

Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect.


Kalyuga, S., Chandler, P., & Sweller, J. (1998). Levels of expertise and instructional design.

Human Factors, 40(1), 1-17.

Kee, D. W., & Davies, L. (1988). Mental effort and elaboration: A developmental analysis.

Contemporary Educational Psychology, 13, 221-228.

Khoo, G.-s., & Koh, t.-s. (1998). Using visualization and simulation tools in tertiary science

education [Electronic Version]. The Journal of Computers in Mathematics and Science

Teaching, 17(1), 5-20.

Lee, J. (1999). Effectiveness of computer-based instructional simulation: A meta analysis.

International Journal of Instructional Media, 26(1), 71-85.

Leemkuil, H., de Jong, T., de Hoog, R., & Christoph, N. (2003). KM Quest: A collaborative

Internet-based simulation game. Simulation & Gaming, 34(1), 89-111.

Locke, E. A., & Latham, G. P. (1990). A theory of goal setting and task performance.

Englewood Cliffs, NJ: Prentice Hall.

Locke, E. A., Latham, G. P. (2002, Summer). Building a practically useful theory of goal setting

and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705-717.

Malone, T. W. (1981). What makes computer games fun? Byte, 6(12), 258-277.

104


Malone, T. W., & Lepper, M. R. (1987). Making leraning fun: A taxonomy of intrinsic

motivation for learning. In R. E. Snow & M. J. Farr (Eds.). Aptitute, learning, and

instruction: Vol. 3. Conative and affective process analyses (pp. 223-253). Hillsdale, NJ:

Lawrence Erlbaum.

Malouf, D. (1987-1988). The effect of instructional computer games on continuing student

motivation. The Journal of Special Education, 21(4), 27-38.

Mayer, R. E. (1981). A psychology of how novices learn computer programming. Computing

Surveys, 13, 121-141.

Mayer, R. E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving.

Instructional Science, 26, 49-63.

Mayer, R. E., & Chandler, P. (2001). When learning is just a click away: Does simple user

interaction foster deeper understanding of multimedia messages? Journal of Educational

Psychology, 93(2), 390-397.

Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When

presenting more material results in less understanding. Journal of Educational

Psychology, 93(1), 187-198.

Mayer, R. E., Mautone, P., & Prothero, W. (2002). Pictorial aids for learning by doing in a

multimedia geology simulation game. Journal of Educational Psychology, 94(1), 171-

185.

Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence of

dual processing systems in working memory. Journal of Educational Psychology, 90(2),

312-320.

105


Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning.


Mayer, R. E., Moreno, R., Boire, M., & Vagge, S. (1999). Maximizing constructivist learning

from multimedia communications by minimizing cognitive load. Journal of Educational

Psychology, 91(4), 638-643.

Mayer, R. E., & Wittrock, M. C. (1996). Problem-solving transfer. In D. C. Berliner & R. C.

Calfee (Eds.), Handbook of educational psychology (pp. 47-62). New York: Simon &

Schuster Macmillan.

McGrenere, J. (1996). Design: Educational electronic multi-player games—A literature review

(Technical Report No. 96-12, the University of British Columbia). Retrieved from

http://taz.cs.ubc.ca/egems/papers/desmugs.pdf

Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of

modality and contiguity. Journal of Educational Psychology, 91(2), 358-368.

Moreno, R., & Mayer, R. E. (2000a). A coherence effect in multimedia learning: The case of

minimizing irrelevant sounds in the design of multimedia instructional messages. Journal

of Educational Psychology, 92(1), 117-125.

Moreno, R., & Mayer, R. E. (2000b). Engaging students in active learning: The case for

personalized multimedia messages. Journal of Educational Psychology, 92(4), 724-733.

Moreno, R., & Mayer, R. E. (2002). Learning science in virtual reality multimedia environments:

Role of methods and media. Journal of Educational Psychology, 94(3), 598-610.

Mousavi, S. Y., Low, R., & Sweller, J. (1995). Reducing cognitive load by mixing auditory and

visual presentation modes. Journal of Educational Psychology, 87(2), 319-334.

106


Mwangi, W., & Sweller, J. (1998). Learning to solve compare word problems: The effect of

example format and generating self-explanations. Cognition and Instruction, 16(2), 173-

199.

Niemiec, R. P., Sikorski, C., & Walberg, H. J. (1996). Learner-control effects: A review of

reviews and a meta-analysis. Journal of Educational Computing Research, 15(2), 157-

174.

Noyes, J. M., & Garland, K. J. (2003). Solving the Tower of Hanoi: Does mode of presentation

matter? Computers in Human Behavior, 19, 579-592.

O’Neil, H. F., Jr. (1999). Perspectives on computer-based performance assessment of problem

solving: Editor’s introduction. Computers in Human Behavior, 15, 255-268.

O'Neil, H. F., Jr. (2002). Perspective on computer-based assessment of problem solving [Special

Issue]. Computers in Human Behavior, 18(6), 605-607.

O’Neil, H. F., Jr., & Abedi, J. (1996). Reliability and validity of a state metacognitive inventory:

Potential for alternative assessment. Journal of Educational Research, 89, 234-245.

O’Neil, H. F., Jr., Baker, E. L., Fisher, J. Y.-C. (2002, August 31). A formative evaluation of ICT

games. Manuscript: University of California, Los Angeles, National Center for Research

on Evaluation, Standards, and Student Testing (CRESST)

O’Neil, H. F., Jr., & Herl, H. E. (1998). Reliability and validity of a trait measure of self-

regulation. Manuscript: University of California, Los Angeles, National Center for

Research on Evaluation, Standards, and Student Testing (CRESST).

Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent

developments. Educational Psychologist, 38(1), 1-4.

107


Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003). Cognitive load

measurement as a means to advance cognitive load theory. Educational Psychologist,

38(1), 63-71.

Park, O.-C., & Gittelman, S. S. (1995). Dynamic characteristics of mental models and dynamic

visual displays. Instructional Science, 23, 303-320.

Perkins, D. N., & Salomon, G. (1989). Are cognitive skills context bound? Educational

Researcher, 18, 16-25.

Pintrich, P. R., & DeGroot, E. V. (1990). Motivational and self-regulated learning components

of classroom academic performance. Journal of Educational Psychology, 82, 33-40.

Porter, D. B., Bird, M. E., & Wunder, A. (1990-1991). Competition, cooperation, satisfaction,

and the performance of complex tasks among Air Force cadets. Current Psychology:

Research & Reviews, 9(4), 347-354.

Quilici, J. L., & Mayer, R. E. (1996). Role of examples in how students learn to categorize

statistics word problems. Journal of Educational Psychology, 88(1), 144-161.

Ramsberger, P. F., Hopwood, D., Hargan, C. S., & Underhill, W. G. (1983), Evaluation of a

spatial data management system for basic skills education. Final Phase I Report for Period

7 October 1980 - 30 April 1983 (HumRRO FR-PRD-83-23). Alexandria, VA: Human

Resources Research Organization.

Randel, J. M., Morris, B. A., Wetzel, C. D., & Whitehill, B. V. (1992). The effectiveness of

games for educational purposes: A review of recent research. Simulation & Games, 23,

261-276.

108


Renkl, A., & Atkinson, R. K. (2003). Structuring the transition from example study to problem

solving in cognitive skill acquisition: A cognitive load perspective. Educational

Psychologist, 38(1), 13-22.

Renkl, A., Atkinson, R. K., Maier, U. H., & Staley, R. (2002). From example study to problem

solving: Smooth transitions help learning. The Journal of Experimental Education, 70(4),

293-315.

Resnick, H., & Sherer, M. (1994). Computerized games in the human services--An introduction.

In H. Resnick (Ed.), Electronic Tools for Social Work Practice and Education (pp. 5-16).

Bington, NY: The Haworth Press.

Ricci, K. E. (1994, Summer). The use of computer-based videogames in knowledge acquisition

and retention. Journal of Interactive Instruction Development, 7(1), 17-22.

Ricci, K. E., Salas, E., & Cannon-Bowers, J. A. (1996). Do computer-based games facilitate

knowledge acquisition and retention? Military Psychology, 8(4), 295-307.

Rieber, L. P. (1996). Seriously considering play: Designing interactive learning environments

based on the blending of microworlds, simulations, and games. Educational Technology

Research and Development, 44(2), 43-58.

Rieber, L. P., & Matzko, M. J. (Jan/Feb 2001). Serious design for serious play in physics.

Educational Technology, 41(1), 14-24.

Rieber, L. P., Smith, L., & Noah, D. (1998, November/December). The value of serious play.

Educational Technology, 38(6), 29-37.

Rosenorn, T., & Kofoed, L. B. (1998). Reflection in learning processes through

simulation/gaming. Simulation & Gaming, 29(4), 432-440.

109


Ruben, B. D. (1999, December). Simulations, games, and experience-based learning: The quest

for a new paradigm for teaching and learning. Simulations & Gaming, 30, 4, 498-505.

Ruddle, R. A., Howes, A., Payne, S. J., & Jones, D. M. (2000). The effects of hyperlinks on

navigation in virtual environments. International Journal of Human-Computer Studies,

53, 551-581.

Ruiz-Primo, M. A., Schultz, S. E., and Shavelson, R. J. (1997). Knowledge map-based

assessment in science: Two exploratory studies (CSE Tech. Rep. No. 436). Los Angeles,

University if California, Center for Research on Evaluation, Standards, and Student

Testing (CRESST).

Salas, E., Bowers, C. A., & Rhodenizer, L. (1998). It is not how much you have but how you use

it: Toward a rational use of simulation to support aviation training. The International

Journal of Aviation Psychology, 8(3), 197-208.

Salomon, G. (1983). The differential investment of mental effort in learning from different

sources. Educational Psychology, 18(1), 42-50.

Santos, J. (2002, Winter). Developing and implementing an Internet-based financial system

simulation game. The Journal of Economic Education, 33(1), 31-40.

Schau, C. & Mattern, N. (1997). Use of map techniques in teaching applied statistics courses.

American statistician, 51, 171-175.

Schraw, G. (1998). Processing and recall differences among seductive details. Journal of

Educational Psychology, 90(1), 3-12.

Shyu, H.-y., & Brown, S. W. (1995). Learner-control: The effects of learning a procedural task

during computer-based videodisc instruction. International Journal of Instructional

Media, 22(3), 217-230.

110


Spiker, V. A., & Nullmeyer, R. T. (n.d.). Benefits and limitations of simulation-based mission

planning and rehearsal. Unpublished manuscript.

Stewart, K. M. (1997, Spring). Beyond entertainment: Using interactive games in web-based

instruction. Journal of Instructional Delivery, 11(2), 18-20.

Story, N., & Sullivan, H. J. (1986, November/December). Factors that influence continuing

motivation. Journal of Educational Research, 80(2), 86-92.

Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and

Instruction, 12, 185-233.

Tarmizi, R. A., & Sweller, J. (1988). Guidance during mathematical problem solving. Journal of

Educational Psychology, 80(4), 424-436.

Tennyson, r. D., & Breuer, K. (2002). Improving problem solving and creativity through use of

complex-dynamic simulations. Computers in Human Behavior, 18(6), 650-668.

Thiagarajan, S. (1998, Sept/October). The myths and realities of simulations in performance

technology. Educational Technology, 38(4), 35-41.

Thomas, P., & Macredie, R. (1994). Games and the design of human-computer interfaces.

Educational Technology, 31, 134-142.

Thompson, L. F., Meriac, J. P., & Cope, J. G. (2002, Summer). Motivating online performance:

The influence of goal setting and Internet self-efficacy. Social Science Computer Review,

20(2), 149-160.

Thorndyke, P. W., & Hayes-Roth, B. (1982). Differences in spatial knowledge acquired from

maps and navigation. Cognitive Psychology, 14, 560-589.

Tuovinen, J. E., & Sweller, J. (1999). A comparison of cognitive load associated with discovery

learning and worked examples. Journal of Educational Psychology, 91(2), 334-341.

111


van Merrienboer, J. J. G., Clark, R. E., & de Croock, M. B. M. (2002). Blueprints for complex

learning: The 4C/ID-model. Educational Technology Research & Development, 50(2),

39-64.

van Merrienboer, J. J. G., Kirschner, P. A., & Kester, L. (2003). Taking a load off a learner’s

mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5-13.

Wainess, R., & O’Neil, H. F. Jr. (2003, August). Feasibility study: Video game research

platform. Manuscript: University of Southern California.

Washbush, J., & Gosen, J. (2001, September). An exploration of game-derived learning in total

enterprise simulations. Simulation & Gaming, 32(3), 281-296.

West, D. C., Pomeroy, J. R., Park, J. K., Gerstenberger, E. A., Sandoval, J. (2000). Critical

thinking in graduate medical education. Journal of the American Medical Association,

284(9), 1105-1110.

Westbrook, J. I., & Braithwaite, J. (2001). The Health Care Game: An evaluation of a heuristic,

web-based simulation. Journal of Interactive Learning Research, 12(1), 89-104.

Westerman, S. J. (1997). Individual differences in the use of command line and menu computer

interfaces. International Journal of Human-Computer Interaction, 9(2), 183-198.

White, R. W. (1959). Motivation reconsidered: The concept of competence. Psychological

Review, 66, 297-333.

Wiedenbeck, S., & Davis, S. (1997). The influence of interaction style and experience on user

perceptions of software packages. International Journal of Human-Computer Studies, 46,

563-588.

112


Wolfe, J. (1997, December). The effectiveness of business games in strategic management

course work [Electronic Version]. Simulation & Gaming Special Issue: Teaching

Strategic Management, 28(4), 360-376.

Wolfe, J., & Roge, J. N. (1997, December). Computerized general management games as

strategic management learning environments [Electronic Version]. Simulation & Gaming

Special Issue: Teaching Strategic Management, 28(4), 423-441.

Yair, Y., Mintz, R., & Litvak, S. (2001). 3D-virtual reality in science education: An implication

for astronomy teaching. Journal of Computers in Mathematics and Science Teaching,

20(3), 293-305.

Yeung, A. S., Jin, P., & Sweller, J. (1997). Cognitive load and learner expertise: Split-attention

and redundancy effects in reading with explanatory notes. Contemporary Educational

Psychology, 23, 1-21.

Yu, F.-Y. (2001). Competition within computer-assisted cooperative learning environments:

Cognitive, affective, and social outcomes. Journal of Educational Computing Research,

24(2), 99-117.

Yung, A. S. (1999). Cognitive load and learner expertise: Split attention and redundancy effects

in reading comprehension tasks with vocabulary definitions. Journal of Educational

Media, 24(2), 87-102.

Zimmerman, B. J. (1994). Dimensions of academic self-regulation: A conceptual framework for

education. In D. H. Schunk, & B. J. Zimmerman (Eds.), Self-regulation of learning and

performance (pp. 3-21). Hillsdale, NJ: Erlbaum.

Zimmerman, B. J. (2000). Self-efficacy. An essential motive to learn. Contemporary Educational

Psychology, 25(1), 82-91.

113


APPENDIX A

Study Examining Problem-SolvingWith Video Games!!!

Participants Wanted!

College students or graduates, 18 years or older, with no experience playing the video game SafeCracker®, needed to participate in a University of Southern California Research Study on Problem-Solving Tasks Using A Computer-Based Video Game.

Study takes approximately 90 minutes.Participants are paid $15.

If you are interested, please contact:Richard Wainess

Ph. D. StudentUniversity of Southern California

[email protected]

114


APPENDIX B

Demographic Information

User ID: _______________________

1. Gender: Male Female

2. Age: _______Years_______Months

3. Average amount per week you play video games (please check one) None 1-2 hours 3-6 hours more than 6 hours

4. On a scale from 1 to 5, rate how much you like playing each game type. One is lowest and five is highest, If you’re unsure or if you don’t know a term, enter a 0 (zero) for the game. Be sure to enter a number for every game type.

a. Puzzle Games _______ (1 to 5; or zero for “don’t know”)

b. RTS Games _______

c. FPS Games _______

d. Strategy Games _______

e. Role Playing Games _______

f. Arcade Games _______

g. PC Games _______

h. Console Games _______

115


APPENDIX C

Task Completion Form (task 1)

User ID: _______________________

Mark the box for each safe you opened during each game.

FIRST GAME: (reception, small showroom, technical design room)

Small Showroom

Blue Safe

Brown Safe

Gray Safe

Technical Design Room

Blue Safe

Strongbox (file cabinet in storeroom)

116


Task Completion Form (task 2)

User ID: _______________________

Mark the box for each safe you opened during each game.SECOND GAME: (Chief engineers room, constructor office, technical design room)

Constructor’s Office

Liberty Safe

Chief Engineer’s Office

Green Safe

Archive

Technical Design Room

Blue Safe

Strongbox (file cabinet in storeroom)

117


APPENDIX D

Self-Regulation Questionnaire

Name (please print): __________________________________________________________________

Directions: A number of statements which people have used to describe themselves are given below. Read each statement and indicate how you generally think or feel on learning tasks by marking your answer sheet. There are no right or wrong answers. Do not spend too much time on any one statement. Remember, give the answer that seems to describe how you generally think or feel.

Almost

Never Sometimes Often

Almost

Always

1. I determine how to solve a task before I begin. 1 2 3 4

2. I check how well I am doing when I solve a

task.

1 2 3 4

3. I work hard to do well even if I don't like a task. 1 2 3 4

4. I believe I will receive an excellent grade in

courses.

1 2 3 4

5. I carefully plan my course of action. 1 2 3 4

6. I ask myself questions to stay on track as I do a

task.

1 2 3 4

7. I put forth my best effort on tasks. 1 2 3 4

8. I’m certain I can understand the most difficult

material presented in the readings for courses.

1 2 3 4

118


Almost


Almost

Always

9. I try to understand tasks before I attempt to

solve them.

1 2 3 4

10. I check my work while I am doing it. 1 2 3 4

11. I work as hard as possible on tasks. 1 2 3 4

12. I’m confident I can understand the basic

concepts taught in courses.

1 2 3 4

13. I try to understand the goal of a task before I

attempt to answer.

1 2 3 4

14. I almost always know how much of a task I

have to complete.

1 2 3 4

15. I am willing to do extra work on tasks to

improve my knowledge.

1 2 3 4

16. I’m confident I can understand the most

complex material presented by the teacher in

courses.

1 2 3 4

17. I figure out my goals and what I need to do to

accomplish them.

1 2 3 4

18. I judge the correctness of my work. 1 2 3 4

119


Almost


Almost

Always

19. I concentrate as hard as I can when doing a task. 1 2 3 4

20. I’m confident I can do an excellent job on the

assignments and tests in courses.

1 2 3 4

21. I imagine the parts of a task I have to complete. 1 2 3 4

22. I correct my errors. 1 2 3 4

23. I work hard on a task even if it does not count. 1 2 3 4

24. I expect to do well in this course. 1 2 3 4

25. I make sure I understand just what has to be

done and how to do it.

1 2 3 4

26. I check my accuracy as I progress through a

task.

1 2 3 4

27. A task is useful to check my knowledge. 1 2 3 4

28. I’m certain I can master the skills being taught

in courses.

1 2 3 4

29. I try to determine what the task requires. 1 2 3 4

30. I ask myself, how well am I doing, as I proceed

through tasks.

1 2 3 4

120


Almost


Almost

Always

31. Practice makes perfect. 1 2 3 4

32. Considering the difficulty of courses, teachers,

and my skills, I think I will do well courses.

1 2 3 4

Copyright © 1995, 1997, 1998, 2000 by Harold F. O’Neil, Jr.

121


APPENDIX E

SafeCracker® Expert Map

122


APPENDIX F

Knowledge Map Specifications

General Domain

Specification

This Software

Scenario Create a knowledge map on the content understanding of

SafeCracker, a computer puzzle-solving game.

Participants

College students or graduate students. Each works on his/her own

knowledge map about SafeCracker, after both the first time and

the second time of playing the game.

Knowledge map

concepts/nodes

Fifteen predefined key concepts identified in the content of

SafeCracker, by experts of the game and knowledge map

professionals. The fifteen predefined concepts are: book, catalog,

clue, code, combination, compass, desk, direction, floor plan,

key, room, safe, searching, trial-and-error, and tool.

Knowledge map

links

Seven predefined important links of relationships identified in the

content of SafeCracker by experts of the game and knowledge

map professionals. The seven predefined links are: causes,

contains, leads to, part of, prior to, requires, and used for.

Knowledge map

domain/content:

SafeCracker

SafeCracker is a computer puzzle-solving game. There are over

50 rooms with about 30 safes; each safe is a puzzle to solve. To

solve the puzzles, players need to find out clues and tools hidden

123


in the rooms, deliberate and reason out the logic and sequence,

try to apply what they have found.

Training of the

computer knowledge

mapping system

All students will go through the same training session.

The training included the following elements:

• How to construct a knowledge map using the computer

mapping system

• How to play SafeCracker, which is the target domain/content

of the programmed knowledge mapper.

Type of knowledge

to be learned

Problem solving

Three problem

solving measures

1. Knowledge map used to measure content understanding and

structure, including (a) semantic content score; (b) the

number of concepts; and (c) the number of links

2. Domain specific problem-solving strategy questionnaire,

including questions to measure problem-solving retention

and transfer

3. Trait self-regulation questionnaire used to measures the four

elements of trait self-regulation: planning, self-checking,

self-efficacy, and effort

124

a formative evaluation of the training effectiveness of a ...€¦ · web viewgoal orientation...

Documents