adapt: a predictive cognitive model of user visual

ADAPT: A Predictive Cognitive Model of UserVisual Attention and Action Planning

STEPHANIE M. DOANE1 and YOUNG WOO SOHN2

1Department of Psychology, P.O. Box 6161, Mississippi State University, Mississippi State,MS 39762, USA, E-mail: [email protected] of Connecticut, Department of Psychology, 406 Babbidge Road, U-20, Storrs, CT,USA

(Received 6 November 1998; accepted in revised form 27 October 1999)

Abstract. We present a computational cognitive model of novice and expert aviation pilotaction planning called ADAPT that models performance in a dynamically changing simulated£ight environment. We perform rigorous tests of ADAPT's predictive validity by comparingthe performance of individual human pilots to that of their respective models. Individual pilotswere asked to execute a series of £ightmaneuvers using a £ight simulator, and their eye ¢xationsand control movements were recorded in a time-synched database. Computational models ofeach of the 25 individual pilotswere constructed, and the individualmodels simulated executionof the same £ightmaneuvers performedby the humanpilots. The time-synched eye ¢xations andcontrol movements of individual pilots and their respective models were compared, and rig-orous tests of ADAPT's predictive validity were performed. The model explains and predictsa signi¢cant portion of pilot visual attention and control movements during £ight as a functionof piloting expertise. Implications for adaptive training systems are discussed.

Key words: cognitive models, action planning, modeling expertise, hybrid models.

1. Introduction

The present research uses a theoretically-based computational model of cognition toexplain and predict pilot behaviors during simulated £ight maneuvers as a functionof £ight situation and piloting expertise. In the present context, planning and actiontake place in tandem and the user (in this case a pilot) is working on multiple tasksat any given moment. Thus accomplishing tasks in the present research refers tothe successful prioritization of multiple goals and actions in a dynamically changingtask environment.

There are numerous theories of how cognitive processes constrain performance inproblem solving tasks, and several have been implemented as computational models.In SOAR, problem solving is constrained by the organization or `chunking' of resultsof searches through memory (e.g. Rosenbloom, Laird, Newell, & McCarl, 1991).Anderson's ACT-R theory assumes that interpretive analogical processes constrainproblem solving performance (e.g. Anderson, 1993). Alternatively, our theoreticalpremise is that comprehension-based mechanisms identical to those used to under-

User Modeling and User-Adapted Interaction 10: 1^45, 2000. 1# 2000 Kluwer Academic Publishers. Printed in the Netherlands.

stand a list of words, narrative prose, and algebraic word problems constrain prob-lem solving episodes as well.

Our premise rests on Kintsch's (1988, 1998) construction-integration theory ofcomprehension. Speci¢cally, Kintsch's (1988) theory presumes that low-level associ-ations between incoming contextual information (e.g. task instructions) and back-ground knowledge (e.g. domain knowledge) are constructed and used toconstrain knowledge activation via a constraint-based integration process. Theresulting pattern of context-sensitive knowledge activations is referred to as a situ-ation model and represents the current state of comprehension. The present researchexamines the predictive validity of the claim that comprehension-based processesplay a central role in cognition (e.g. Doane, Sohn, Adams, & McNamara, 1994;Gernsbacher, 1990; Kintsch, 1988, 1998; Schmalhofer & Tschaitschian, 1993;van Dijk & Kintsch, 1983). Proponents of this view have proposed detailed cognitivemodels of comprehension (e.g. Kintsch, 1988; Doane, Sohn, McNamara & Adams,in press), and provided evidence for the importance of comprehension for under-standing cognition in general (e.g. Gernsbacher, 1990).

Kintsch's theory in particular has been used to explain a wide variety of behavioralphenomena, including narrative story comprehension (Kintsch, 1988), algebra storyproblem comprehension (Kintsch, 1988), the solution of simple computing tasks(Mannes & Kintsch, 1991), and completing the Tower of Hanoi task (Schmalhofer& Tschaitschian, 1993). This approach has also proved fruitful for understandinghuman-computer interaction skills (e.g. Doane, McNamara, Kintsch, Polson, &Clawson, 1992; Kitajima & Polson, 1995; Mannes & Doane, 1991) and predictingthe impact of instructions on computer user performance (Doane et al., in press;Sohn & Doane, 1997). The breadth of application suggests that the comprehensionprocesses described in Kintsch's model play a central role in many tasks, and assuch it may be considered a general architecture of cognition (Newell, 1987; Kintsch,1998).

The present study extends the Construction^Integration theory of comprehensionto account for user cognition in the complex and dynamically changing environmentof airplane piloting. Speci¢cally, we evaluate whether ADAPT, a construction^integration model of piloting skill can predict the focus of pilot visual attentionand control manipulation during simulated £ight maneuvers. This was accomplishedby simulating the performance of twenty-¢ve actual pilots on twelve segments of£ight, and then comparing human and model performance data to determineADAPT's predictive validity.

This extension makes three important contributions. First, we are testing theability of a cognitive theory to predict visual information gathering and multipletask performance in an applied task environment. Such tests are necessary to supportthe centrality of comprehension-based processes beyond the controlled laboratoryenvironment. Second, we test the ability of the theory to predict user visual focus,an aspect of cognition detached from many models of problem solving (Meyer& Kieras, 1997; but see Anderson & Douglass, in press). Third, we performed rig-

2 STEPHANIE M. DOANE AND YOUNG WOO SOHN

orous tests of the predictive rather than descriptive validity of our computationaltheory. The present study represents a signi¢cant methodological contributionfor evaluation of computational models of human cognition (e.g. Thagard, 1989;Doane et al., in press). Computational cognitive models of operator behavior inreal-time environments have in the past been plagued by ad hoc explanations ofhow task-sensitive knowledge activation occurs. The present e¡ort demonstratesthat ADAPT can predict visual information gathering and action planning perform-ance for multiple pilots during simulated £ight maneuvers.

In the following we provide a brief summary of our rationale for choosingthe comprehension-based approach as our framework. We then describeADAPT, and the experiments that tested the ability of ADAPT to predict pilotperformance.

1.1. CONSTRUCTION^INTEGRATION THEORY

The construction-integration model (Kintsch, 1988) was initially developed toexplain certain phenomena of text comprehension, such as word sense dis-ambiguation. The model describes how we use contextual information to assigna single meaning to words that have multiple meanings. For example, the appro-priate assignment of meaning for the word `bank' is different in the context of con-versations about paychecks (money `bank') and about swimming (river `bank').In Kintsch's view, this can be explained by representing memory as an associativenetwork where the nodes in the network contain propositional representations ofknowledge about the current context or task, general (context-independent)declarative facts, and If/Then rules that represent possible plans of action (Mannes&Kintsch, 1991). The declarative facts and plan knowledge are similar to declarativeand procedural knowledge contained in ACT-R (e.g. Anderson, 1993).

When the model simulates comprehension in the context of a speci¢c task (e.g.reading a paragraph for a later memory test), a set of weak symbolic productionrules construct an associative network of knowledge interrelated on the basis ofsuper¢cial similarities between propositional representations of knowledge withoutregard to task context. This associated network of knowledge is then integratedvia a constraint-satisfaction algorithm that propagates activation throughout thenetwork, strengthening connections between items relevant to the current task con-text and inhibiting or weakening connections between irrelevant items. This inte-gration phase results in context-sensitive knowledge activation constrained byinter-item overlap and current task relevance.

The ability to simulate context-sensitive knowledge activation is most importantfor the present work. We are studying the construction of adaptive, novel plansof action rather than studying retrieval of known routine procedures (e.g. Holyoak,1991). Symbolic/connectionist architectures such as the one utilized in the presentstudy use symbolic rules to interrelate knowledge in a network, and then spreadactivation throughout the network using connectionist constraint-satisfactionalgorithms. This architecture has signi¢cant advantages over solely symbolic or con-

PREDICTING USER ACTION PLANNING 3

nectionist forms for researchers interested in context-sensitive aspects of adaptiveproblem solving (e.g. Broadbent, 1993; Holyoak, 1991; Holyoak & Thagard, 1989;Mannes & Doane, 1991; Thagard, 1989).

1.2. MODELING PROBLEM SOLVING PERFORMANCE

van Dijk and Kintsch (1983) and Kintsch (1988; 1994) suggest that comprehendingtext that describes a problem to be solved (e.g. an algebra story problem) involvesretrieving relevant factual knowledge, utilizing appropriate procedural knowledge(e.g. knowledge of algebraic and arithmetic operations), and formulating a solutionplan.

Kintsch's framework has successfully modeled interactive problem solving tasks.Using a computational construction/integration model called UNICOM, Doaneet al. (1992; in press) modeled command production performance and the skill acqui-sition of UNIX users while they interacted with an instructional tutor (UNIX is acommand-line based computer operating system). In UNICOM, instructional textand the current state of the operating system serve as cues for activation of the rel-evant knowledge and for organizing this knowledge to produce an action sequence.The focus of the analysis was not so much on understanding the text per se, buton the way these instructions activated the UNIX knowledge relevant to the per-formance of the speci¢ed task.

1.3. OTHER APPROACHES TO MODELING OPERATOR BEHAVIOR IN REAL-TIME

ENVIRONMENTS

It is appropriate at this point to compare the present approach to other viableapproaches currently in use. The argument overlap used by ADAPT differs fromthe structure of models such as COGNET (e.g. Zachary, 1996) and MIDAS (e.g.Hoecker et al., 1994). The attention switching model in COGNET is inspired bySelfridge's pandemonium model (Selfridge, 1988), and a preset value is usually usedto resolve attentional con£icts. It is also interesting to note that the tasks inCOGNET consist of whole procedures and strategies that can be interrupted byhigher priority tasks. COGNET must resume the interrupted procedure at somepoint with variables that re£ect the new situation. This problem is avoided inADAPT because whole procedures are not represented in a precompiled form; rathershort procedural sequences are dynamically concatenated together as the problemsolving situation warrants.1

It is also important to mention that the application of comprehension-basedtheories to real-time problems also has historical precedence. The Defense AdvancedResearch Projects Agency (DARPA) initiated a Pilot's Associate program todevelop a decision support system for ¢ghter aircraft pilots (e.g. Rouse, Geddes,

1We thank an anonymous reviewer for suggesting this interesting contrast.


& Curry, 1988). The core of this system consisted of a hierarchical plan-goal graphthat described how di¡erent aspects of pilot expertise a¡orded goal accomplishmentthroughout a £ight mission. The development of the graph was inspired by earlyresearch on language understanding (e.g. Wilensky, 1978; Schank & Abelson, 1977).Unfortunately, the pilot model was seen by some operational personnel as toosimplistic, and this has hampered development e¡orts. Other research programshave included more sophisticated cognitive models of pilots, but the models havebeen so speci¢c that their validity out of speci¢c task contexts has been questioned(e.g. Rouse, Geddes & Curry, 1988). Because the architecture of ADAPT has beentheoretically validated in numerous problem solving environments, it does not su¡erfrom the speci¢city problem which plagued the Pilot Associate Program. However, itremains to be seen whether operational personnel will accept ADAPT in, forexample, tutor form.

In the present research, we model the visual attention and £ight performance oftwenty-¢ve individual pilots during simulated £ight maneuvers. Prior to discussingthe architecture of the ADAPT model and the modeling experiments, we will sum-marize the £ight simulation studies that measured individual pilot's eye movementsand £ight performance during £ight maneuvers.

2. Empirical Study of Pilot Visual Scanning and Flight Performance

The purpose of this study was to track the eye movements of novice and expertprivate aviation pilots while they completed £ight maneuvers during simulated £ight.Results from this study were used to build and test our ADAPT model of pilot visualattention and £ight performance. The eye-movement study was part of a largerproject and a joint effort with additional collaborators. The results from this studyare only summarized here. Further details are available in Fox et al. (1995) andin Doane and Sohn (1999).

METHOD

2.1. PARTICIPANTS

Twenty-¢ve participants, student pilots and instructor pilots, from the Instituteof Aviation at the University of Illinois participated in the study. Allparticipants who participated in the experiment did so voluntarily and received$5 per hour.

A pre-experimental questionnaire was administered to determine the level ofpilots' £ight experience. Twenty-two nonredundant questions were submitted toa principal-components factor analysis (varimax rotation). The resulting organi-zation of questions into factors was used to classify participants into expertisegroups. The analysis produced two factors with eigen values greater than 1 account-ing for 57% of the variance in the data. The ¢rst factor, labeled `general experience,'


accounted for 45% of the variance. This factor included questions about certi¢cateratings (private certi¢cate, commercial certi¢cate, and certi¢ed £ight instructor),£ight time on a simulated instrument, training time in an approved £ight simulator,the number of £ight courses taken, types of airport £own into, and the numberof di¡erent multi-engine aircraft £own. The second factor, labeled `practicalexperience,' accounted for 12% of the variance. This factor included variables about£ight time on an actual instrument, instrument £ight time within the last 90 days, andthe number of di¡erent single-engine aircraft £own.

We conducted a discriminant analysis (using jackknife procedures) on the basis ofthe two factors to classify participants into expertise groups for the purpose ofanalysis. As a result of the analysis, the participants were classi¢ed into threeexpertise groups: Novice, intermediate, and expert groups contained eight, eleven,and six participants, respectively. Novices had an average total £ight time of49 hr and an average number of £ight courses of 2.5; Intermediates had an averageof 481 hr and an average of 8.3 courses; Experts had an average of 1467 hr andan average of 9.8 courses.

2.2. APPARATUS

2.2.1. Flight Simulator

A Gateway 2000 computer with an SVGA graphics card produced the £ightsimulation. A color 19-inch monitor displayed the instrument panel with the stan-dard primary instruments (the airspeed indicator, attitude indicator, altimeter, turncoordinator, heading indicator, vertical speed indicator [VSI]), and tachometer)and a cockpit instruction panel that contained a clock and a bird's-eye view ofthe seven £ight segments (see Figure 1). The instruments depicted in the displaywere 6� in diameter with a minimum separation of 2.2�. The cockpit instructionpanel contained boxes that displayed where pilots should `Start,' `Change to,'and `Finish' a maneuver. The `Start' box speci¢ed the heading, altitude and airspeeddesired at the beginning of a maneuver. The `Change to' box speci¢ed the £ightmaneuver (e.g. change altitude to 1650 ft.) The `Finish' box displayed the desiredheading, altitude, and airspeed of the plane at the end of the £ight maneuver. Pilotsused a sidearm-mounted joystick to provide control inputs such as roll (lateral stickmovement), pitch (fore-aft stick movement), and power (push or pull back move-ment of a button atop the stick). The joystick was placed to the right of the displayscreen, and stick inputs were sampled at 5 Hz.

2.2.2. Head-mounted Eye Tracker

Eye scan measures were made using an Applied Sciences laboratory series 4100Hhead-mounted eye tracker. Sampling and output rate of the tracking camera was60 Hz.


Figure1.

Exa

mple£igh

tsimulator

cockpitdisplay.

Depictedare7£igh

tinstruments

includ

ingtheairspeed

indicator,attitude

indicator,altimeter,tacho

meter,

turn

coordina

tor,

head

ingindicator,

andvertical

speed

indicator(V

SI).

Also

depicted

arebo

xedtables

show

ingtheexperimentalinstructions,£igh

tpa

th,

andelap

sedtimein

thesegm

ent.


2.3. PROCEDURE

The experiment was run over two days. On the ¢rst day participants spentapproximately 25 min familiarizing themselves with the control dynamics ofthe simulation environment and the eye tracker. On the second day, participantspracticed £ying seven different £ight segments that required airspeed, heading,and/or altitude changes (see Figure 2). The ¢rst three segments of the simulationrequired a change in only one £ight axis; airspeed, heading, or altitude (theother two were to be maintained at starting values). The fourth through sixthsegments required changes in two £ight axes, and the seventh segment requiredthree axes changes. Following a short break, the same seven £ight segmentswere £own again and this time eye-movement and £ight performance data werecollected.

One possible criticism of this experiment is that subjects performed the same tasksto familiarize themselves with the simulator as they did later when data wasrecorded. That is, one might argue that this introduces an uncontrolled variableto the experiment, namely learning di¡erences between individuals. If, for example,we had varied the order of tasks from familiarization to testing, proponents of thisview might argue that learning would be restricted to the simulator operationwhereas in the present study it pertains to both the simulator operation and tothe particular tasks at hand.

For the present purposes we are assuming that individual di¡erences inlearning ability during familiarization trials did not negatively impact the abilityof the ADAPT model to predict human pilot behavioral data. Since ADAPTis not modeling learning, this is an interesting yet somewhat peripheral pointunless one assumes that the problem with the familiarization trials was that theyled to a ceiling e¡ect and decreased the heterogeneity in the data and essentiallyeased the tasks of ¢tting a model to data. However, performance data didnot indicate a ceiling e¡ect, and the heterogeneity in the data was su¤cientto result in signi¢cant systematic variability in the data as a function of expertiseand £ight situation complexity, as reported in later sections (also see Doane etal., 1999).

Each segment was preceded by a 30-second straight and level lead-in. Thisallowed participants time to read the segment instructions displayed in a smallinstruction panel displayed on screen. The changes for each £ight segment werestipulated in the aforementioned instruction panel and could be accomplishedwithin 60 sec in the ¢rst ¢ve segments and within 75 sec in the last two segments.If the participants were not within �50 feet of altitude, �5 degrees of heading,and �5 knots of airspeed at the end of the allotted time frame, the segmentwas ended by the computer and the participants were placed at the beginningof the next lead-in leg. This procedure enabled the recording of seven independentsegments of £ight performance for each subject. The time required to completeall the seven segments was about 11 min.


Segment 1:Airspeeddecrease

Segment 2:Heading to the right

Segment 3:Altitudeincrease

Segment 4:Airspeed increase &Heading to the left

Segment 5:Airspeed decrease & Altitude decrease

Segment 6:Heading to the right &Altitude increase

Segment 7:Airspeed increase &Heading to the left &Altitude decrease

START

STOP

Figure 2. Graphical depiction of the seven £ight segments £own by individual pilots using a £ight simulator. The double solid lines mark the beginning of a segment.Each segment begins with a 30 sec lead-in, the end of which is depicted by a single dashed line. The lead-in is followed by either a 60 sec £ight segment (segments 1^5)or 75 sec £ight segment (segments 6^7) where eye ¢xations and control movements are recorded.

PREDIC

TIN

GUSE

RACTIO

NPL

ANNIN

G9

2.3. PILOT PERFORMANCE DATA

Figures 3(a)^(c) depict the three time-synched measures of pilot and plane perform-ance recorded; eye-scans, control movements, and plane status. The eye scanmeasures (Figure 3(a)) indicate eye ¢xations (eye movement holding at a point)and dwells (time period during which a ¢xation or a series of continuous ¢xationsremain within an area of interest) on each of the 7 £ight instruments. Control move-ment measures (see Figure 3(b)) indicate left-to-right (bank control) and fore-to-aft(pitch control) joystick inputs and fore-to-aft (power control) movements of a buttonatop the stick that participants used to control power. Plane performance measures(see Figure 3(c)) indicate the heading, altitude, and airspeed status (including currentvalues and changes in progress) of the simulated airplane.

The example data shown in Figure 3 indicates that at the start of the segment thepilot was looking at the airspeed indicator (Figure 3(a)), while pulling back onthe throttle (Figure 3(b)), and that this input resulted in a plane airspeed decrease(see Figure 3(c)). Four seconds into the segment, the pilot ¢xated on the attitudeindicator, and 1.5 sec later ¢xated on the VSI, etc.

3. Simulation Experiments

The data just described were used to measure the match between actual andmodeled pilot performance. Prior to describing our modeling experiments, wepresent the details of the ADAPT model of pilot skill. We begin by describingthe ADAPT architecture, and then describe the procedures used to represent theknowledge of individual pilots that participated in the empirical study. This isfollowed by a description of the methods used to simulate pilot and planeperformance. Finally, comparisons between modeled and actual pilot performanceare presented.

3.1. ADAPT KNOWLEDGE REPRESENTATION

ADAPT represents human memory as an associative network where each prop-osition representing knowledge constitutes a single node. In the discussion ofADAPT's architecture we use, for the most part, planning terms as de¢ned by Allenand Perrault (1980). The main exceptions are the use of the terms `outcome,' which inthe present work refers to the cognitive consequence of performing a mental actionand `in the world knowledge' which in the present paper refers to a cognitive under-standing of the transient state of the world. That is, `world' knowledge in the presentcontext represents a pilot's understanding of their current £ight goals and planestatus as well as a memory of recent events. It does not re£ect the knowledge ofa hardware simulator. Rather, it represents the often inaccurate and partial knowl-edge of the human operator. In ADAPT, each proposition contains a predicateand some number of arguments, and in ADAPT they represent knowledge aboutthe piloting domain or the present task. For example, propositionalization of


Joystick(Right)

(Left)

Throttle(Back)

(Forward)

Joystick(Forward)

(Back)

1 2 3 4 5 6 7 8 9 100 (sec.)

1 2 3 4 5 6 7 8 9 100 (sec.)

Heading

Altitude

Airspeed

+50

-50

+5

-5

1 2 3 4 5 6 7 8 9 100 (sec.)

Heading Indicator

Turn Coordinator

Altitude Indicator

Altimeter

VSI

Tachometer

Airspeed Indicator

(a) Eye Scans

(b) Control Movements

(c) Plane Performance

-><

-->

-><

-<

-

5 Left

5 Right

Figures 3(a)^(c). Example time-synched empirical recordings of eye ¢xations, control movements, andplane performance for the ¢rst 10 sec of Segment 1.


the sentence, `(The pilot) knows pulling back the throttle decreases power' wouldappear as (know pull back throttle decrease power). Though we do not use atomicpropositions, we follow the methods outlined by Turner and Green (1978).

ADAPT represents the three major classes of knowledge proposed by Kintsch(1988); world knowledge, general knowledge (e.g. declarative facts), and planelement knowledge (e.g. procedural knowledge represented as if/then rules).Table 1 shows examples of each class of knowledge along with their abbreviatedpropositional representation in ADAPT.

3.1.1. World Knowledge

The ¢rst class of knowledge represents the modeled pilot's current state of the world.Examples of world knowledge in ADAPT include the pilot's knowledge of the cur-rent and desired states of the airplane, determined relationships between the currentand desired states (e.g. altitude is higher than desired value), and £ight segmentgoals. Airplane status is represented as the current values, rates of change, and direc-tion of change of each £ight axis. Current values represent the current status of theairplane shown on display instruments (e.g. 3000 ft), direction of change representsincrease, decrease, or holding axis status, and rate of change represents the speedof change to an axis (e.g. 500 ft per min). These facts are contextually sensitiveand £uid, changing as the modeled task and simulated performance progressesthrough a £ight maneuver. If the ADAPT `pilot' detects a change in the state ofthe airplane, then plane status is updated, and the relationships between currentand desired plane states recalculated. In addition, if the ADAPT `pilot' receivesnew £ight goals, then goals are updated in the modeled `pilot' world knowledge.

It is important to note that ADAPT constrains the number of propositions thatcan be retained in world knowledge to crudely model working memory limitations.Working memory capacity limitations are modeled by limiting the number of pro-positions retained in world knowledge from cycle to cycle using a predetermined`capacity' threshold. Decay limitations are modeled by limiting the length of time(modeled as `cycles') a given in-the-world proposition can be retained in memoryusing a predetermined `decay' threshold. The speci¢c procedures used to determinethreshold values and to remove propositions that exceed threshold values aredescribed below in the procedures section.

3.1.2. General Knowledge

The second class of knowledge, general knowledge, refers to factual (declarative)knowledge about piloting (see Table 1). In ADAPT, general knowledge representsfacts about the relationships between control inputs and plane performance, andknowledge of £ight dynamics, display instruments, and control movements. Theminimum number of general knowledge propositions included in a modeled pilot'sknowledge base was 38, and the maximum number included was 64.


3.1.3. Plan Element Knowledge

The third class of knowledge, plan elements, represent èxecutable' (procedural)knowledge about piloting. Plan elements describe actions that can be takenin-the-world, and they specify conditions under which actions can be taken. Thus,pilots have condition-action rules that they can consider and execute if conditionsare correct. Plan elements are three-part knowledge structures that include a name,preconditions in-the-world or general knowledge that must be satis¢ed in orderfor the plan to ¢re, and outcomes that are added to world knowledge if the planis executed (see Table 1). For example, as shown in Table 1, a plan element thatdecreases airspeed requires that the pilot know the desired airspeed is less thanthe current airspeed, that power controls airspeed, the tachometer indicates power,and pulling back the throttle decreases power. When the decrease plan elementis ¢red, its outcome propositions are added to the world knowledge to represent

Table I. Examples of knowledge and their formal representations in the ADAPT model.

Type of Knowledge Abbreviated Propositional Representation

World KnowledgeDesired airspeed is 90 kts (Know airspeed^desired=90)Current airspeed is 100 kts (Know airspeed^current=100)Desired airspeed is much less thancurrent airspeed

(Know much^lt-airspeed^desired-airspeed^

current)A task is to check airspeed (Task check airspeed)

General KnowledgeControl-performance relationshipPower controls airspeed (Know power control airspeed)

Flight dynamicsPitch up causes airspeed decrease (Know pitch up airspeed decrease)

Primary-supporting displayVSI supports altimeter (Know VSI support altimeter)

Display instrumentAirspeed indicator indicates airspeed (Know airspeedîndicator indicate airspeed)

Control movementPulling back throttle decreases power (Know pull^back throttle decrease power)

Plan KnowledgeName:Decrease airspeed (Do decrease airspeed)

Preconditions:Desired airspeed is less than currentairspeed

(Know lt-airspeed^desired-airspeed^current)

Airspeed indicator indicates airspeed (Know airspeedîndicator indicates airspeed)Power controls airspeed (Know power control airspeed)Pulling back throttle decreases power (Know pull^back throttle decrease power)

Outcome(s):Need to look at airspeed indicator (Need airspeedîndicator)Need to pull back throttle (Need pull^back throttle)


the current need to look at the tachometer and to pull back the throttle.2 The mini-mum number of plan elements included in a modeled pilot's knowledge basewas 98 and the maximum was 122.

3.2. ARCHITECTURE OF PLAN ELEMENTS

This section describes the types of plan elements represented in ADAPT and detailstheir theoretical interrelationships. Table II shows detailed examples of plan elementknowledge. As shown in the table, the model contains two major classes of planelements: cognitive and action. There are three forms of cognitive plan elements(i.e. monitor-status, determine, and change) that represent mental operations orthought processes hypothesized to motivate explicit behaviors (e.g. eye-scansand control movements). Two forms of action plan elements (i.e. monitor-displayand control plans) represent explicit pilot behaviors (i.e. the eye-scans and controlmovements measured in the empirical study of individual pilots).

3.2.1. Types of Plan Elements

3.2.1.1. Monitor-Status. The monitor-status plan elements check the current statusof a particular plane £ight axis. The outcomes of this plan element are preconditionsof an action monitor-display plan element that represents eye scans to a displayinstrument. For example, once the `monitor airspeed' plan (see the monitor-statusplan in Table 2) ¢res, its outcome `need to look at airspeed indicator' will be addedto the world. This is a precondition that must exist in-the-world before the `lookat airspeed indicator' plan element (see the monitor-display plan in Table 2) can¢re. This precondition is necessary but not suf¢cient for the `look at airspeedindicator' plan element to ¢re, as detailed below. When and if the `look at airspeedindicator' plan element ¢res, the current airspeed will be updated in-the-world.

3.2.1.2. Determine. The determine plan element calculates the relationship betweenthe desired and current states of the plane. For example, the plan to determinethe relationship between the desired and current airspeed (see the determine planin Table 2) requires the desired and current values of airspeed to exist in-the-worldbefore it can ¢re. As shown in Figure 4, the determine plan element is enabledby an outcome of the monitor-display plan that updates the current state of theairplane. When the determine plan element is ¢red, the updated relationshipsbetween current and desired states of the aircraft will be added to the worldknowledge. In Table 2 the outcome ¢eld of the determine plan element is dependenton the result of calculating the difference between current and desired values ata particular C/I cycle. If the difference between the desired and current states does

2The full set of ADAPTpiloting knowledge and all other details ofmodeling procedures includ-ing scoring rules, plane behavior rules, and rules for synchronizing human andmodeled dtat areavailable on line at http:wildthing.psychology.msstate.edu/


not exceed the predetermined limit (e.g. within � or ÿ50 ft of the desired altitude),the calculation determines that they are `equal.' If the difference between the currentand desired states exceeds the predetermined limit (e.g. more than 50 ft above or 50 ftbelow the desired altitude), then the calculation determines that the current value is`greater than' or `less than' the desired value depending on the direction of thedifference, and execution of a change plan element is enabled. By enabled we meanthat the state of the world that exists when the difference between the currentand desired states exceeds predetermined limits is necessary for a change planelement to ¢re. However, such a state is not suf¢cient for the plan element to ¢re.

Table II. Examples of plan knowledge. Italicized text indicates an outcome calculated on the basisof the state of the world when the plan element is ¢red.

Cognitive PlansExample Monitor-Status PlanName: Monitor airspeedPreconditions: Task is to check airspeed

Know airspeed indicator indicates airspeedOutcome: Need to look at airspeed indicator

Example Determine PlanName: Determine relationship between desired and current airspeedPreconditions: Looked at airspeed indicator

Know current airspeedKnow desired airspeed

Outcome: Know desired airspeed is {less than, equal to, higher than}current airspeed

Example Change PlanName: Decrease airspeedPreconditions: Know desired airspeed is less than current airspeed

Know airspeed indicator indicates airspeedKnow power controls airspeedKnow pulling back throttle decreases power

Outcomes: Need to look at airspeed indicatorNeed to pull back throttle

Action PlansExample Monitor-Display PlanName: Look at airspeed indicatorPreconditions: Need to look at airspeed indicator

Know airspeed indicator indicates airspeedOutcomes: Looked at airspeed indicator

Know current airspeed

Example Control PlanName: Pull back throttlePreconditions: Need to pull back throttle

Know pulling back throttle decreases powerOutcomes: Pulled back throttle


As shown in Figure 4, if the determine plan element ¢res and `determines' the dif-ference between the current and desired states is within threshold limits, thenthe model will continue monitoring the status of the airplane, performing supportingchecks within the same £ight axis or a cross check across the different £ight axes untilthe need for a change is detected.

3.2.1.3. Change. The change plan element, if ¢red, sets a cognitive goal of chang-ing the status of a £ight axis. This plan can be executed when the current axis statusis not within desired limits. The execution of this plan can be followed by amonitor-display plan that represents an eye scan action and/or a control planelement that represents a speci¢c control movement (see Figure 4). For example,once the `decrease airspeed' plan (see the change plan in Table 2) is ¢red, itsoutcomes, `need to look at the airspeed indicator' and `need to pull back throttle'are added to the world. These are preconditions that must exist in-the-world beforethe `look at airspeed indicator' and `pull back throttle' plan elements (see the controland monitor-display plans in Table 2) can ¢re. After the control movement is made,monitor display plan elements can ¢re to check main- and/or side-effects of thecontrol change executed.

3.2.2. Relationships Between Cognitive and Action Plan Elements

Figure 4 depicts the theoretical relationships between the plan element types. Asshown in the ¢gure, precondition and outcome mappings between plans enable

Monitor-Status Plan

Monitor-Display Plan Determine Plan

Change Plan

Control Plan

Figure 4. Structure of plans in ADAPT.


the execution a `sequence-dependent' set of plans. By enable we mean that the ¢ringof one plan can result in an outcome that serves as a precondition for another planelement. As such, the outcome of the former plan element is necessary for the latterplan element to ¢re. However, the outcome is not suf¢cient for the latter planelement to ¢re. That is, in ADAPT as in other planning systems it is necessaryfor preconditions to be met before a plan element can ¢re; but in ADAPT thisis not suf¢cient for a plan element to ¢re. Multiple plan elements must competefor the greatest amount of activation at each C/I cycle, and working memorylimitations may result in outcomes being deleted from world knowledge before aplan whose precondition is satis¢ed by that outcome can ¢re. For example, the ¢ringof the `monitor altitude' plan element may place the outcomes `need altimeter' and`need VSI' in world knowledge. However, the presence of these `need' propositionsin the world knowledge will not guarantee the subsequent ¢ring of the `look ataltimeter' or the `look at VSI' plan elements even though the `need' propositionssatisfy preconditions necessary for these `look at' plan elements to ¢re. Either orboth of the `look at' plan elements could fail by unsuccessfully competing with otherplan elements for activation in subsequent C/I cycles, or they could fail if the `need'outcomes that serve as preconditions fall out of working memory on a givenC/I cycle because the number of `need' propositions in the world knowledge exceedworking memory capacity. To summarize, although precondition/outcomemappingin£uences plan activation, other factors in£uence plan element activation and thisplus working memory limitations ensure that the sequence of plan elementexecutions are not `hard wired' or ¢xed.

3.2.2.1. Degrees of freedom in plan element ¢ring. In an effort to quantify the degreesof freedom in the sequence of plan elements ¢red by ADAPT, we will list the optionsavailable to ADAPT at each C/I cycle. To foreshadow the result, the degrees offreedom range from 16 to 46 plan elements whose preconditions are met andcan be ¢red at any given C/I cycle. The exact number at a given C/I cycle dependson the modeled pilot's working memory capacity and the extent to which the airplaneperformance attributes deviate from desired values.

3.2.2.2. Degrees of freedom in cognitive plan elements. Each of the three types ofcognitive plan elements has distinct degrees of freedom. First, at any given C/I cycle,there are three monitor-status plan elements that can be ¢red, because theirpreconditions remain in the world throughout the simulation. Second, determineplan elements (which `determine' if a plane performance attribute is within desiredlimits) have as a precondition that `trace' outcomes exist in the world which representthe current and the desired values of the relevant plane performance attribute (e.g.airspeed). ADAPT models of individual pilots have working memory limitationsthat delete these `trace' propositions when working memory is exceeded. The rangeof `trace' propositions allowed in working memory for individual pilots modeledby ADAPT ranges from 4 to 7. However, there is not a one-to-one correspondence


between `trace' outcomes and determine plan elements. For example, the `trace' ofthe attitude indicator current value can serve as a precondition for the determinepitch plan element and for the determine bank plan element. As a result, eventhough there is a limit of 4^7 `trace' propositions being held in working memoryin ADAPT, the number of determine plan elements that can ¢re at a given C/Icycle is greater than 4^7. Calculations of the number of determine plan elementswith preconditions satis¢ed by `trace' propositions translate into between 4 and16 determine plan elements that can be ¢red at any given C/I cycle. Third, changeplan element ¢ring is enabled if their corresponding plane performance attributes(e.g. airspeed, heading, altitude) are not within the desired limits. For example,if the determine plan element ¢res and indicates that the current airspeed is lowerthan the desired airspeed and that the difference exceeds a predetermined thresholdvalue, this is necessary for the change airspeed plan element to ¢re (though notsuf¢cient). There are a maximum of 24 distinct change plans that can ¢re dependingon the relationships between current and desired values for airspeed, heading,altitude, power, bank, and pitch. If the airplane is not within desired limits forany of these attributes, then there are 6 change plan elements that could ¢re ata given C/I cycle. If the airplane is within desired limits for all of the attributes,then none of the change plan elements would be able to ¢re.

In sum, the degrees of freedom for cognitive plan elements ¢ring at a given C/Icycle ranges from a minimum of 7 cognitive plan elements (3 monitor-status planelements, 4 determine plan elements, and 0 change plan elements) to a maximumof 25 cognitive plan elements (3 monitor-status plan elements, 16 determine planelements, 6 change plan elements).

3.2.2.3. Degrees of freedom in action plan elements. The action plan elements havepreconditions for `need' propositions to be present in the world knowledge. The`need' propositions are used as preconditions for 7 monitor-display plan elementsand for 12 control plan elements (e.g. pull-back throttle). There is a one- to-onecorrespondence between `need' propositions in the world and action plan elements.That is, a `need' proposition will only satisfy the preconditions of one action planelement. As previously stated, preconditions are necessary for plan elements to ¢re,but they are not suf¢cient. Thus, although there is a one-to-one correspondencebetween the `need' propositions and the action plan elements, the existence ofthe former does not guarantee the ¢ring of the latter.

As was true for `trace' propositions, there are working memory limitationsADAPT uses to determine how many `need' propositions can remain in workingmemory. To model individual di¡erences in working memory capacity for `need'propositions, ADAPT allowed a range of 9^21 `need' propositions to be retainedin world knowledge. (The procedures for assigning working memory limitationsfor individual pilot models are described in a later section.) Capacity limitationswere enforced by deleting the `need' propositions with the lowest activation fromworking memory when their number exceeded capacity.


3.2.2.4. Summary of plan element degrees of freedom. ADAPT uses an executiverule to ¢re plan elements which considers the most activated plan element whosepreconditions are met in the world or in general knowledge. There are no restrictionsin ADAPT to constrain the type of plan element that must be ¢red on a given C/Icycle. Therefore it is appropriate to sum the degrees of freedom for cognitiveand action plan elements to determine total degrees of freedom. The total degreesof freedom for the ¢ring of cognitive and action plan elements ranges from 16^46at any given C/I cycle.

This discussion of degrees of freedom is important because it suggests thatADAPT is not deterministic in the sense of having a ¢xed set of plan elementspreset to ¢re. These relatively large degrees of freedom suggest that ADAPT's¢t to human data (described below) would not be easily accounted for by, forexample, a simple recency rule. Furthermore, comparisons between ADAPTand strictly rule-based models exist in the literature, (see, e.g. Mannes &Doane, 1991) and they suggest signi¢cant advantages of ADAPT's hybridsymbolic/connectionist architecture for modeling context-dependent planningand action (also see, Kintsch, 1998). In addition, ADAPT uses as its frameworka cognitive theory of comprehension that has been used to explain human behaviorin various problem solving contexts (see, e.g. Kintsch, 1998), and as such it furthersour understanding of the role of comprehension in complex task performance.Alternatives such as recency rules and many strictly rule-based models do not pro-vide a generalizable theoretical account of human cognition in complex taskenvironments.

3.2.3. Representations of Cognitive and Action Plan Elements

Many factors in£uenced the plan element representations in this model. First, inorder to devise a model that could be rigorously tested, we de¢ned plan elementswhose execution by subjects could be veri¢ed by observation of pilot's eye ¢xations,control movements, and plane performance. To de¢ne a ¢ner granularity wouldresult in a larger number of unobservable elements and would decrease our abilityto rigorously test the model's ability to predict human performance.

Second, we wanted to account for discrepancies between pilot cognitive goalsand their actual performance. For example, pilots in the present studies know thatpiloting requires monitoring the status of various attributes of plane performance(e.g. airspeed, altitude, heading). However, pilots will sometimes fail to lookat displays at critical periods during £ight. In fact, pilot training includespracticing instrument `scans' in an e¡ort to train pilots to continually update theirawareness of information displayed by each cockpit instrument. Despite thistraining, critical failures in scan patterns occur. To account for this discrepancybetween piloting goals and actual performance, the plan elements described belowrepresent the goal of monitoring an airplane performance attribute separate fromthe actual plan element to accomplish the goal. Thus, some plan elements will


have similar preconditions, such as `monitor-airspeed' and `look-at airspeedindicator.' However, their separation is necessary in order to di¡erentiate the acti-vation of a goal to monitor a plane performance attribute and activating thebehavior (i.e. eye movement) that will provide the updated status information.It is important to note that a given pilot may show variability in the successor failure of completing the chain of goal-to-action plan elements dependingon the state of the world at a given time. For example, novice pilots often showa breakdown in scanning when their workload is high. Our separate representationof goal and action plan elements and ADAPT's dynamic planning architectureallow the representation of such within-pilot failures as a function of £ightsituation.

Third, the separate representation of goal and action plan elements allows us tomodel variability in actions pilots choose as a function of £ight situation. Forexample, a pilot may use di¡erent actions to increase altitude in one £ight situationversus another. Speci¢cally, when the desired altitude is greater than the currentaltitude and altitude is not changing, a pilot may `look at' the attitude indicatorand the altimeter and initiate an elevator control movement to increase altitude.In contrast, when the desired altitude is greater than the current altitude andthe altitude is increasing at a lower rate of change than desired, the pilot may scanthe attitude indicator and the VSI while initiating an elevator control movementto increase the rate of ascent. In both cases the pilot has `determined' that the currentaltitude is less than desired and that the discrepancy is beyond threshold limits.However, the speci¢c circumstances of the £ight situation led to di¡erent actionsto accomplish the same goal, that of increasing altitude.

As previously stated, the fact that a plan element calls for a scan of instrumentsdoes not guarantee that they will be scanned. The model may scan none, one orboth, depending on working memory limitations and competition from other planelements for activation. In summary, even within individual pilots scanning patternsvary as a function of £ight situation. For reason, cognitive and action plans need tobe separated to model the failure of action even when a goal has been activatedto accomplish that action.

3.3. CONSTRUCTING INDIVIDUAL KNOWLEDGE BASES

3.3.1. Procedure to Create Expert Knowledge Base

We created a `prototypical' expert knowledge base of necessary £ight knowledgeusing various sources. Initially, we used pilot training manuals (e.g. Dogan, 1991;Federal Aviation Administration, 1980) to create a set of £ight rules. Five pilotinstructors modi¢ed and evaluated our initial set of £ight rules for accuracy andcompleteness. Knowledge bases for individual pilots contained some or all ofthe prototypical expert knowledge.


3.3.2. Constructing Individual Knowledge Bases

Twenty-¢ve individual knowledge bases were constructed to represent the individualpilots that participated in the empirical study. The contents of each pilot's knowledgebase were determined through observations of a small portion of the pilot's eye-scan,control movement, and airplane performance data.

3.3.2.1. Observing samples of individual pilot performance data. Data were sampledfrom six different 7^15 sec time blocks (`windows') for each individual pilot, asshown in Table 3. Thus, we sampled 56 sec of empirical performance data to buildindividual knowledge bases that were then used to `predict' approximately 11min of pilot behavior during simulated £ight maneuvers. The sampling windowswere chosen to score pilot knowledge while initiating, maintaining, and ¢nishing£ight maneuvers.

3.3.2.2. Scoring pilot knowledge based on sampled performance data. Wescored missing knowledge using an overlay method (see VanLehn, 1988), anddevised explicit knowledge scoring rules. Knowledge scoring rules were used

Table III. Empirical data windows sampled to score individual pilot's knowledge.

WindowSamplingBlock

AverageNumber ofEye-ScansSampled

AverageLengthof TimeSampled

Knowledgeto be Scored

1 Beginning ofSegment 1

7 7 sec Establishing airspeed changeMonitoring consequences of powercontrol

2 Middle ofSegment 1

20 15 sec Eye-scan patternsHolding cruise airspeed

3 End ofSegment 2

7 7 sec Rolling out of heading changeMonitoring consequences of releasingbank control


10 10 sec Establishing altitude changeMonitoring consequences of pitchcontrolMonitoring consequences of multiplecontrols

5 End ofSegment 3

7 7 sec Leveling o¡ altitude changeMonitoring consequences of releasingpitch control


10 10 sec Control changes for multiple tasksMonitoring consequences of multiplecontrols

Establishing heading changeMonitoring consequences of bankcontrol


to determine if pilots possessed speci¢c types of knowledge of Control-Performance Relationships, Flight Dynamics, Primary and Supporting Displays,Display Instruments, Control Movement, Cross-Axis Check, Main Effects, and SideEffects and procedures toMonitor Status, Change Status, and Determine Status. Wedevised a total of 28 £ight knowledge scoring rules which were applied to evaluate theperformance observed in the performance sampling window for each pilot and deter-mine what would be included in their knowledge base.

In this example, the rule evaluates whether performance indicates knowledge thatthe airspeed indicator displays change in airspeed initiated by throttle movement.The abbreviate rule states:

IF while making a control change, the pilot looks at a particular instrumentthat displays the plane's response to that change,

THEN the pilot knows that the instrument scanned displays the change estab-lished by the control movement.

In the 0^4 sec section of Figure 3, the pilot looks at the airspeed indicator whilepulling back the throttle. This behavior satis¢es the IF portion of the scoring ruleshown above, and as a result this pilot's knowledge base would include knowledgethat the airspeed indicator displays airspeed change. This fact was representedin general knowledge as the proposition `know airspeed indicator indicates airspeed.'It was also represented as a decrease airspeed plan element whose outcome includedthe proposition `need to look at airspeed indicator.'

4. ADAPT Simulations

4.1. OVERVIEW OF MODEL EXECUTION

For plans to be selected for execution, they must be relevant to the current anddesired £ight status (as dictated by the world knowledge) and their preconditionsmust exist in the knowledge base. The model operates in a cyclical fashion, ¢ringthe most activated plan element whose preconditions exist in the knowledge base.The outcome proposition(s) of the ¢red plan elements are added to the world,and construction begins again with the modi¢ed knowledge base. Following con-struction, the modi¢ed and associated knowledge is integrated and a subsequentplan element is selected for execution. This process continues until the goals areaccomplished. As previously mentioned, plan elements are not selected in a ¢xedsequence to accomplish a single goal. In contrast, the modeled pilot is workingon multiple goals at a time and multiple plan elements compete for activation. Thusthe model may ¢re a plan element relevant to goal1 in a given C/I cycle, and then¢re a plan element relevant to goal2, and then a plan element relevant to goal1. There are no apriori constraints on the order of plan element ¢ring in ADAPT.In the following section we will detail the modeling procedures used within a Con-struction/Integration (C/I) cycle, and then between C/I cycles.


4.2. PHASES EXECUTED WITHIN A SINGLE C/I CYCLE

Knowledge about the domain and a particular task is represented in a distributedmanner although the node content remains symbolic and identi¢able. It is the pat-tern of activation across nodes that determines the current model of the problemsituation. The following sections detail how these symbolic nodes are interrelated.

4.2.1. Construction

During construction, ADAPT computes relationships between propositions in theknowledge base (k) to construct a task-speci¢c network of associated knowledge

W

G

P

Knowledge base including General and Plan knowledge and the desired and current status of the plane in the World

x [11...1 0 0 0 ... 0]

Task connectivity matrix

Initial activation vector

[2.3 0.9 1.5 ... 0.7]

W

G

P

W G P

Resulting activation vector

Is change in successive vector activation < .0001?

Normalize resulting activation vector and post multiply by initial activation vector to continue integration process.

Yes

No

Most recent resulting activation vector becomes the final, integrated activation vector.

(1) Access knowledge base

(2) Construct an associated knowledge network

in-the-world knowledge

(3) Multiply the task connectivity matrix by the initial activation vector

(4) Produce a resulting activation vector

(5) Continue integration process until activation of knowledge is stabilized

(6) Produce a final activation vector

Figure 5. Schematic representation of ADAPT computations during one construction/ integration cycle(W�World knowledge; G � General knowledge; P�Plan knowledge).


as depicted in Figure 5, steps 1^2. The model uses low-level rules to construct asymmetric task connectivity matrix (c), where each node (c(i,j)) contains a numericvalue corresponding to the calculated strength of the relationship between k(i)and k(j). Equations for the strength calculations are detailed in Section A.1 ofAppendix A. The resulting network, depicted in Figure 5, represents theunconstrained relationships between knowledge brought to bear to accomplish thisspeci¢c task. The low-level rules used to determine if two nodes were related didnot vary between or within simulations. In fact, the inter-node relationships havenot changed since Kintsch's (1988) model introduction.

4.2.1.1. Associative and semantic relationships. Associative and semantic relation-ships between each pair of propositional nodes (c(i,j)) in the network are basedon the number of shared arguments, and completely embedded propositions. Pro-positions are linked with a positive weight for each argument shared. For example,the propositions (know power control airspeed) and (do increase airspeed) shareone argument (airspeed). The corresponding nodes in the network (c(i,j); c(j,i))would be positively linked with a weight of 0.4 because they share this argument.If one proposition is entirely embedded in another, the two propositions are linkedwith a weight of 0.8. Although these relationships provide only a crude approxi-mation of propositional relatedness, they have been effective in prior simulations(Doane et al., in press; Kintsch, 1988; Miller & Kintsch, 1980).

4.2.1.2. Plan element relationships. Although plan elements contain three ¢elds(i.e. name, precondition, outcome), they are represented as a single node in thenetwork. Only the name ¢eld of a plan element is included in the aforementionedcalculations of semantic and associative relatedness.

Overlap between plan element precondition and outcome ¢elds is calculated toestimate causal relationships between plan elements. For example, if the outcome(s)of one plan (p(j)) satisfy the precondition(s) of another (p(i)), then a positive asym-metric weight of 0.7 will be added to the respective c(i,j) node in the task connectivitynetwork. Functionally this allows the activation of p(i) to £ow to p(j) duringintegration. If an outcome(s) of one plan element negates the precondition(s) ofanother, then an asymmetric inhibitory link of ÿ10.0 is entered into the correspond-ing c(i,j) node.

Figure 6 depicts these causal relations for three abbreviated example plan elementsto increase, decrease, and determine power. In this example, a positive link existsfrom `Increase power' to `Determine power' because the outcome of the Determinepower plan element satis¢es the `know desired power greater than current power'precondition of the Increase power plan element. An inhibitory interplan relationfrom `Decrease power' to `Determine power' exists because the outcome of thisexample is `know desired power greater than current power' of the Determine powerplan element functionally negates the Decrease power precondition `know desiredpower less than current power.'


Two relations between plans and world knowledge are calculated. First, if theoutcome(s) of a plan element already exists in-the-world, then an asymmetric inhibi-tory link of ÿ10.00 exists between each `in-the-world' proposition that matches theoutcome(s) propositions. For example, if the outcome of the increase power planelement `need to push forward throttle' exists in-the-world, the increase power planelement is inhibited during integration. Another related inhibitory link of ÿ0.4is used to relate traces representing actions previously accomplished (e.g. Trace pullback throttle) and name propositions of plans that will accomplish the alreadyexecuted goal (e.g. Execute pull back throttle) that share an argument overlap. Thesetwo inhibitory relationships are calculated to keep the model from repeating itself.

Second, if the name or the outcome ¢elds of a plan element match the request andoutcome propositions that represent the current task in-the-world, a positive link of1.5 is made between the matching propositions. Speci¢cally, a symmetric weightof 1.5 is applied to the respective links between matching the request and planelement name propositions, and the outcome and plan element outcome pro-positions.

To summarize, ADAPT uses the construction relationships and weights devisedby Mannes and Kintsch (1991), including argument overlap weights of 0.4, a prop-osition embedding weight of 0.8, plan element precondition and outcome mappings

_

+

Increase powerKnow desired power greater than current powerNeed to push forward throttle

Name:Precondition:

Outcome:

Decrease powerKnow desired power less than current powerNeed to pull back throttle

Name:Precondition:

Outcome:

Determine powerKnow desired powerKnow current powerKnow desired power greater than current power

Name:Precondition:

Outcome:

Figure 6. Example precondition and outcome interplan relationships (`�'� positive; `ÿ'� inhibitory).Italicized text indicates outcome bound to result for the current state of the world.


of 0.7 (positive), and ÿ10.00, and ÿ0.4 (inhibitory), and a weight of 1.5 for theaforementioned request and outcome propositions that match plan element namesand outcome ¢elds, respectively.3 Where no link was speci¢ed, connections wereset to zero. These parameter values have been used and remain constant in allof our construction/integration simulations (see Doane et al., in press). As suggestedby Thagard (1989), this weight stability is critical for assessing the reliability of acognitive architecture across simulation e¡orts.

4.2.2. Integration

The constructed network of knowledge represents unconstrained relations betweenknowledge elements. To develop a situation model (e.g. Kintsch, 1988), this knowl-edge must be integrated by using constraint-based activation to spread activationthroughout the network. This process essentially strengthens the activation ofknowledge elements consistent with the task context and the environmental situationof £ight, and dampens the activation of others. The simple linear algorithm used tointegrate the constructed knowledge base is illustrated in Figure 5, steps 3^6,and formulas are provided in Section A.2 of Appendix A.

Computationally, integration constitutes the repeated post-multiplication of theconstructed network (matrix) by a vector. The vector values represent the currentactivation of each knowledge element represented (e.g. the value of the ¢rst itemin the vector represents the current state of activation of the ¢rst proposition inthe knowledge base, and so on).

As depicted in Figure 5, the initial vector values corresponding to in-the-worldknowledge are set to 1.0, and all others are set to 0. Functionally, this allowsthe in-the-world propositions that represent the current task context and £ight situ-ation to drive the spread of activation. This `initial activation vector' ispost-multiplied by the connectivity matrix resulting from the construction process(see step 3 in Figure 5), and a `resulting activation vector' is produced (see step4 in Figure 5). After each multiplication, the vector weights corresponding tothe current set of in-the-world items are reset to 1.0, and the remaining itemsare normalized to ensure their sum is a constant value across integrations (see step5 in Figure 5).

The iterative integration process stops when the di¡erence between two successiveactivation vectors is less than 0.0001. At this point, the resulting activation vectorbecomes the ¢nal activation vector and represents the stabilized activation ofknowledge. The ¢nal activation vector is then used by the model to make executivedecisions regarding the next plan element to ¢re.

3The exact values of the weights are arbitrary, though their relative values do represent weightsgiven to particular associations in the construction phase.


4.2.3. Plan Selection

The model ¢nds the most activated plan element in the ¢nal activation vector, anddetermines whether its preconditions exist in-the-world or general knowledge. If theyexist, then the plan is selected to ¢re, and its outcome propositions are added to theworld knowledge. If they do not exist, then this process is repeated using thenext-most-activated plan element until a plan can be ¢red.

4.3. UPDATING IN-THE-WORLD KNOWLEDGE BETWEEN-C/I CYCLES

Because we are simulating behavior in a dynamically changing environment, ourmodel must continually update the state of the world. While the computational pro-cedures used to accomplish this task are not intended to represent a cognitive`scratch pad' per se, they need to be mentioned for the purpose of exposition.

4.3.1. Updating Based On Outcomes of Executed Plan

4.3.1.1. Adding plan outcome (trace, need, and know) propositions. As previouslystated, once a plan ¢res outcome propositions are added to world knowledge.Speci¢cally, trace, need, and know outcomes that respectively represent pastactions, desired actions, and the current status of the airplane are added to worldknowledge.

4.3.1.2. Deleting obsolete need and trace propositions. The model deletes the needand trace propositions from the world that are no longer relevant as a result of plan¢ring. If a need proposition in-the-world was satis¢ed by the ¢ring of a plan,the need proposition was deleted from the world. For example, if the `pull backthrottle' plan ¢res, the proposition `need to pull back throttle' is deleted fromthe world. Trace propositions representing prior control inputs are deleted oncemonitor-display plans are ¢red. We assumed that when pilots obtained updatedstatus information on a given £ight axis, they did not need to retain memory ofall previous control inputs.

In a similar sense, if a control plan was ¢red to make a change in an axis, any tracepropositions that represented a monitor of the relevant axis displays would bedeleted from world knowledge. When pilots make a control change, previous tracesof axis status are rendered obsolete.

4.3.1.3. Updating plane status. The status of each £ight axis was automaticallyupdated as the model £ew according to programmed plane behavior rules.Speci¢cally, the status, direction of change, and rate of change were updated foreach £ight axis. The status value indicates the location of the plane on the axis (e.g.3000 for altitude), the direction indicates whether the value is currently increasing,decreasing, or holding (represented numerically as �1, ÿ1, or 0), and the rateindicates how fast the value is changing (e.g. 5 ft per cycle). When no control plan


was ¢red, the current status of the plane at the tth cycle was approximated by addingthe product of the change direction and rate to the status value of the plane at the(tÿ 1)th C/I cycle. For example, if the value, direction, and rate for the altitudeat the 10th cycle were 3000, � 1, and 5 respectively, the altitude value at the 11thcycle would be 3005 (3000� 1� 5) unless a control plan was ¢red.

When a control plan was ¢red, the e¡ect of the change on the plane behavior wasformulated into the equations to update the plane status. To simplify the represen-tation of idiosyncratic control behavior, we classi¢ed individual pilots' controlmovement patterns for pitch, bank, and power changes into three types: correct-,under- and over-shoot. Classi¢cation depended on how the pilot reached the desiredrates of changes in the sampling windows previously mentioned. Pilots that used aninitial single control movement which resulted in obtaining the desired rate of changewere classi¢ed as making `correct-shoot' control movements. Pilots that used mul-tiple small control movements to incrementally reach the desired rate of change wereclassi¢ed as making `under-shoot' control movements. Finally, pilots that used aninitially large control movement which resulted in exceeding the desired rate ofchange followed by small incremental movements to decrease the rate of changewere classi¢ed as making `over-shoot' control movements. In general, pilots wereconsistent in their use of one of the three control movement classi¢cation patternsacross the pitch, power, and bank changes. However, each pilot's model wasassigned a bias score to represent their control movement patterns for pitch, power,and bank changes. These bias scores were incorporated into the plane behavior rules.

For example, if a control plan to increase altitude ¢res at the (tÿ 1)th cycle, thealtitude would be updated at the subsequent tth cycle based on the pilot's controlmovement style as follows:

For correct-shoot,altitudet� altitudetÿ1� standard rate of climb;For under-shoot,altitudet� altitudetÿ1� (altitude direction)tÿ1 (rate of climb)tÿ1�1/2 [(standard rate of climb)ÿ (altitude direction)tÿ1 (rate of climb)tÿ1];

For over-shoot,altitudet� altitudetÿ1� (altitude direction)tÿ1 (rate of climb)tÿ1� 3/2 [(standard rate of climb)ÿ (altitude direction)tÿ1 (rate of climb)tÿ1].

The inter-axis e¡ects were also calculated. For example, ¢ring a control plan toincrease altitude at the (tÿ 1)th cycle may decrease the rate of airspeed changeat the tth cycle.

4.3.2. Updating Based on Memory Constraints

For tasks that require processing and storage resources to retrieve, compare, andstore information, working memory serves an important function (e.g. Chase &Ericsson, 1982; Ericsson & Kintsch, 1995; Just & Carpenter, 1992; Sohn & Doane,


1997). To account for the impact of working memory limitations on the pilotingperformance, we incorporated two memory components in the ADAPT modelto represent capacity and decay constraints.

4.3.2.1. Capacity. Capacity limitations were represented by deleting those needand trace propositions with the lowest activations following a C/I cycle untilthe number retained is less than or equal to the memory capacity parameter value.For example, if the capacity limits are set to 4 and 9 for the trace and need pro-positions respectively, the trace and need propositions that are not among the fourthand ninth most activated respectively are deleted. Because the activation of all pro-positions are constrained by their relevance to the current task context, we aresimulating context-sensitive working memory limitations.

4.3.2.2. Decay. Decay was represented by representing the age of each propositionin-the-world, and incrementing propositional age by one after each con-struction/integration cycle. We used a decay threshold to delete old propositionsfollowing each C/I cycle. For example, if the decay threshold is set at 7, in-the-worldpropositions older than 7 C/I cycles are deleted.

4.4. PROCEDURES USED TO TRAIN AND TEST INDIVIDUAL ADAPT MODELS

Once an individual knowledge base for each pilot was constructed, the model was`trained' by using the initial knowledge base to simulate the ¢rst 10 sec of perform-ance in segment 1. Following training, the model was `tested' by simulating pilotperformance in each of the seven £ight segments included in the empirical studywithout our intervention.

4.4.1 Training

Knowledge-based models have long been criticized as being descriptiverepresentations of unfalsi¢able theories (e.g. Dreyfus, 1992; 1996, D. A. Anderson,1989). In the present work we constructed individual knowledge bases by observinga small portion of empirical performance data. This model was trained usingprincipled rules to correct omissions or commissions of knowledge, and then tunedto represent individual pilots control movement strategies, and working memorycapacity and decay. Once this was accomplished, we used automated modeling pro-cedures to simulate, and thereby predict, the remainder of the observed pilot data(which were not used in constructing the pilot model). This type of procedure ismore commonly used by researchers in machine learning than by researchers thatdevelop cognitive models. However, validations of knowledge-based cognitivemodels are increasing (e.g. Anderson 1993; Doane et al., in press; Lovett &Anderson, 1996; Recker & Pirolli, 1995; Thagard, 1989).


4.4.1.1. Procedure. The training procedure consisted of three procedural stages.First, the initial values for the capacity and decay parameters were set to their lowestpossible values. Second, the ¢rst 10 sec of £ight in segment 1 was simulated withexperimenter intervention following each cycle. Speci¢cally, to facilitate compari-sons between model performance and that of the individual pilot modeled, the stateof the world was reset following each construction/integration cycle. This took placeonly in the training phase. For example, if the ¢rst plan ¢red by the model at thebeginning of segment 1 did not match the eye scans or control movements ofthe individual pilot, the mismatch was recorded and the modeled world statuswas `reset' to values they would be should the model have ¢red a matching plan.That is, the modeled 'world` status was reset to represent the world situation experi-enced by the individual pilot following their ¢rst action.

Third, the knowledge base and parameter values were modi¢ed based on theresults of training. If the model ¢red a plan that did not match the pilot's perform-ance due to lack of knowledge, then the data were reviewed once more to see ifwe missed a demonstration of that knowledge. That is, we checked to see if the itemwas demonstrated in the predetermined sampling windows using the previouslydescribed scoring rules. If a scoring error was detected, the knowledge base wasmodi¢ed accordingly. If not, the knowledge base remained unchanged.

In contrast, if the mismatch between the modeled performance and that of theindividual was due to the retention or deletion of an in-the-world proposition,the decay and capacity memory parameters were adjusted in the appropriatedirection. For example, if the model's capacity parameter was set at 4, and itdid not ¢re the correct plan (e.g. `look at airspeed indicator') because the relevantprecondition (e.g. `need to look at airspeed indicator') was deleted as the 5th mostactivated proposition in the last cycle, then the capacity parameter would beincreased to 5 in the model. That is, if the model is `forgetting' more quickly thanthe pilot during this small training window, we adjusted the parameter to slowthe model rate of forgetting.

These three training steps were repeated until we were sure all possible knowledgehad been scored according to our knowledge scoring rules, and until memory-basederrors were minimized. No further changes were made to the knowledge base orto memory parameters once this training phase was completed.

4.4.2. Group Di¡erences in Modeled Knowledge and Memory Parameters

Although we modeled individual pilots, our analyses will focus on the average ¢tbetween model and individual pilot behaviors as a function of piloting expertise.Thus it is relevant to characterize the relevant group differences in modeled knowl-edge and memory parameters.

4.4.2.1. Group differences in piloting knowledge. The amount of knowledge aboutcontrol-performance relationships (e.g. a power control has an effect on an air-


plane's airspeed), £ight dynamics (e.g. if the power decreases, the altitude woulddecrease), display instruments (e.g. the airspeed indicator indicates the current air-speed), primary-supporting displays (e.g. the VSI supports the altimeter), cross-axescheck (e.g. looking at the altimeter followed by the heading indicator), and controlmovements (e.g. pulling back the throttle increases the power), control display (e.g.looking at the tachometer while making a throttle movement), main-effect check(e.g. looking at the airspeed indicator to check the effect of pulling back throttle),and side-effect check (e.g. looking at the altimeter to check the effect of pulling backthe throttle) contained in each individual pilot model was calculated. Figure 7 showsthe mean knowledge scores on the nine types of knowledge for the three expertisegroups. In general, the only group differences in knowledge apparent are for £ightdynamics, cross-axes check, and display instrument knowledge.

4.4.2.2. Group differences in memory parameter values. Figure 8 shows the meansize of the capacity and decay memory parameters for the three expertise groups.Expert models have a greater capacity parameter value and longer decay thresholdthan the other groups. This is consistent with the theories that working memorydemands differentially impact complex task performance as a function of expertise(cf. Ericsson & Kintsch, 1995; Sohn & Doane, 1997)

0

2

4

6

8

10

12

Am

ount

Knowledge Type

ExpertIntermediateNovice

Control-Performance

FlightDynamics

Primary-SupportingDisplay

DisplayInstrument

ControlMovement

Side-EffectCheck

ControlDisplay

Main -EffectCheck

Cross-AxesCheck

Figure 7. Mean knowledge scores as a function of knowledge types for novice, intermediate, and expertmodels (The error bars indicate standard errors).


4.5. TESTING

4.5.1. Procedure

ADAPT was tested by simulating each pilot's performance on the seven segments of£ight. The testing procedures are schematically summarized in Figure 9. A givenpilot's knowledge base was accessed by ADAPT, and the model was given desired£ight goals and beginning plane status. This information matched what was givento individual pilots at the start of a given £ight segment. The model executed a con-struction and integration cycle, and selected the most activated plan element whosepreconditions were met in either world or general knowledge to ¢re. The outcome(s)

0

5

10

15

20

25

30

Siz

e

Decay Capacity

Memory Components

ExpertIntermediateNovice

Figure 8. Mean size of memory components for novice, intermediate, and expert models (The error barsindicate standard errors).


ARead task instructions :Add starting and finishing status of the plane for the segment to world knowledge.

Access knowledge :Access knowledge base for individual model.

Fire? Are the planpreconditions met in either world or general knowledge?

Yes

No

B

Construct and integrate :Construct then integrate knowledge base. (see Figure 5)

Select plan :Find most activated plan element.

Fire plan : Add outcome(s) of plan element to world knowledge and delete obsolete need and trace propositions from the world.

Look for next most activated plan element.

B

Memory overload?Is there world knowledge beyond memory limits?

Delete current task instructions and outcomes from world knowledge. Move to next segment.

Task completed?Did the plane capture the goal status of the segment?

Yes

No

Yes

No

A

Delete the propositions least activated or older than predetermined cycles.

Figure 9. Schematic representation of procedures used to simulate one pilot's piloting performance.


of the plan element ¢red were then added to the world knowledge and the need andtrace propositions were updated to indicate the revised state of the world.

Following knowledge base revisions, the model determined if working memorycapacity or decay thresholds had been exceeded. If so, the model retained the mostactivated (capacity) and recent (decay) propositions that fell within the limits setfor the individual model during the training phase. This procedure was repeatedfor each segment of £ight until the model obtained the desired £ight goals orexceeded arbitrary time (cycle) limits (approximately 150 cycles). The entire proce-dure was automated.

5. Results and Discussion

5.1. EXAMPLE MODEL PERFORMANCE

The main data of interest is the match between the sequence of actions observed forindividual pilots and the corresponding sequence of plans ¢red by their ADAPTsimulated performance. For the purpose of exposition, we will refer to the actionsobserved for individual pilots as `pilot plans.' Table 4 shows an example comparisonof model and pilot plan ¢ring sequence for the goal of increasing airspeed andturning to the left. As shown in Table 4, the current state of the aircraft is 100 ktairspeed, 180� heading, and 3500 ft altitude. The goal status is 110 kt, 0�, and 3500ft for airspeed, heading, and altitude respectively. The model ¢rst ¢red a cognitivechange plan to increase airspeed, then it ¢red action plans (push forward throttleand look at tachometer) to accomplish the change. Next the plan to determinethe relationship between the desired and the current power ¢red. Thus, the individualpilot actions match the sequence of plans ¢red by the model during the same stages of£ight. As previously mentioned, the cognitive plans were hypothesized to representthe goal-driven behavior of pilots during simulated £ight maneuvers. For example,if a pilot pulled back the throttle while looking at the tachometer and consequentlythe airspeed of the plane was decreased, we hypothesized that the pilot performeda task to decrease airspeed.

Mismatches are depicted in Table 4 as well. For example, the Table indicates themodel ¢red a plan to look at the turn coordinator at cycle 9. This does not matchthe corresponding pilot behavior during the corresponding £ight period. It is import-ant to note that because the model operates in a cyclical fashion, actions are executedserially. The sequential ¢ring of single plans while the £ight status remains constantis treated as a parallel action, though clearly we cannot di¡erentiate immediate suc-cession of serial actions from those that are parallel with the present model.

Another type of mismatch occurs when the modeled and actual pilot executesmatching actions in the opposite order. Looking at cycles 12^13 in Table 4, the model¢res plans to look at the VSI and then the altimeter, while the individual pilotexecuted these behaviors in the opposite order.


5.2. VALIDATION OF ADAPT

5.2.1. Synchronizing Individual Pilot and Pilot Model Performance Data

The match between individual and modeled pilot behavior was calculated only forcomparable £ight situations. Although the status of the actual and modeled £ightsituation matched at the beginning of each segment, we did not intervene to maintainthis match. As a result, if the model executed a plan that was not echoed in theindividual pilot's data, then the modeled and actual £ight situations may differ.Clearly the mismatch of behavior in different £ight situations is not of interest,and as a result the data analyzed represents that of the model and the individualpilot in identical £ight situations. Table 5 shows the average percent time in seconds

Table IV. Example model and pilot performance (Goal: airspeed = 110; heading = 0; altitude = 3500,Current status: airspeed = 100; heading = 180; altitude = 3500)

Fired Model Plans Observed Pilot PlansCycle Cognitive Plan Action Plan Sec. Cognitive Plan Action Plan

1 Increase airspeed 0 Increase airspeed

2 Push forward throttle Push forwardthrottle

3 Look at tachometer Look at tachometer

4 Determine power Determine power

5 Change headingto the left

1.5 Change headingto the left

6 Left pressure onjoystick

Left pressure onjoystick

7 Look at attitudeindicator

Look at attitudeindicator

8 Determine pitchand bank

Determine pitchand bank

9 Look at turncoordinator

10 Determine rateof turn

11 Monitor altitude Monitor altitude

12 Look at VSI 3.2 Look at altimeter

13 Look at altimeter 3.8 Look at VSI

14 Determine altitude Determine altitude

15 Look at attitudeindicator

4.5 Look at attitudeindicator

16 Determine pitchand bank

Determine pitchand bank


scored for individual pilots in each segment of £ight. As the table shows, thevalidation covers 35^44% of each Table V segment's data. While at ¢rst glance thismay seem a small fraction of the entire data set, we know of no other cognitivemodel subjected to such rigorous tests of predictive validity in a complexdynamically changing task environment such as £ight.

5.2.1.1. Flight situation measures. A £ight situation is characterized by threevariables (current status, direction of change, and rate of change) for three £ightaxes (airspeed, heading, and altitude). The current status, direction of change,and rate of change were categorized as within-limits/beyond-limits, increasing/decreasing/holding, and high/low/none, respectively. If the £ight situationvariables for the three axes had the same values for individual and modeled pilotsduring a similar time period within a segment, then their £ight situations were judgedas `comparable' and behavior by the individual pilot and plans ¢red by the modelwere included in data analyses.

5.2.1.2. Coding time. We synchronized the individual pilot data and modeled databy introducing a goal-based unit of processing time called `coding time.' Because theindividual pilot's behavior was measured as a function of time and the model'sbehavior is measured in cycles, we needed to devise a way to synchronize timein seconds with time in cycles. To do this we created a goal-based unit of processingtime that refers to all activities taking place while a particular cognitive goal is active.This coding time increases by one when a cognitive goal is accomplished andremoved from world knowledge. For example, if the cognitive goal to change air-speed is active, all behaviors that take place until the change is accomplishedare considered to take place within the same coding time. Once the change in air-speed is accomplished, the cognitive change plan is removed from world knowledge,and coding time is incremented by one.

If the model or the individual pilot appear to work on two cognitive goals inalternation or in parallel, all actions recorded while this takes place are givenone coding time although two cognitive goals are active. For example, if thecognitive plans to monitor heading and airspeed are both active in-the-world,

Table V. Total £ight time, mean £ight time scored, and mean percent £ight time scored for eachsegment

Total £ight time Flight time scored Percent time scoredSegment (sec) (sec) (%)

1 60 20.8 352 60 24.4 413 60 26.3 444 60 24.0 405 60 26.2 446 75 31.2 427 75 24.7 33


and the individual or modeled pilot appears to work on both plans by repeatingactions necessary for each goal in alternation (or in parallel in the case of the indi-vidual pilot), then the actions for both goals are scored as taking place in the samecoding time. For example, alternating scans to the airspeed indicator and the head-ing indicator while airspeed and heading cognitive plans (monitor or change) areactive would be an indication that the two goals are being worked on at the samecoding time.

5.2.2. Translating Model and Human Actions Into Comparable Plans

Individual pilot performance was related to plans as follows. The monitor-displayplan (e.g. look at the airspeed indicator) represents an eye ¢xation on the particularinstrument. The control plan (e.g. back pressure on joystick) represents anestablishing and releasing control pressure action (e.g. setting back pressure onjoystick plus releasing the pressure back to the neutral position). If the individualpilot made more than two stepped control movements to change a £ight axis,we scored the two largest movements. The model does not simulate control theoreticaspects of manual control.

5.3. MEASURES OF INDIVIDUAL PILOT AND MODELED PILOT PERFORMANCE

5.3.1. Plans

5.3.1.1. Scoring pilot and modeled pilot plan match. To quantify the ¢t between thesimulation and empirical data, percent plan match was calculated on the basis of thereduced performance data as follows:

Percent plan match� (number of matches in plans ¢red between pilot and model in acoding time) � (total number of model plans ¢red in a codingtime)�100

5.3.1.2. Example computation of percent plan match. To illustrate the computationof the percent plan match, we will take the hypothesized data for an individual pilotand individual pilot model in Table 4 as an example. We computed percent matchesfor the cognitive and the action plan types separately. The percent matches for bothtypes of plans in the ¢rst coding time (cycles 1^4) are 100% (i.e. 2/2� 100), wherethe number of matches is 2 and the total number of model plans is 2 for eachof the plan types. The percent matches for the cognitive and action plans are all67% (i.e. 2/3� 100) in the second coding time (cycles 5^10) and all 100% (i.e.3/3� 100) in the third coding time (cycles 11^16). The order of plan execution withina coding time was not considered when counting matches. Recall that, as statedabove, coding time is a goal-based unit of processing time that refers to all activitiestaking place while a particular cognitive goal is active. This coding time increases byone when a cognitive goal is accomplished and removed from world knowledge. For


example, if the model ¢red look at plans for the altimeter and the VSI within thesame coding time as the individual pilot looked at the VSI and the altimeter, thiswas scored as a match. The mean percent plan match from the example data wouldbe 89% for each of the cognitive and action plans that is the average of the percentmatches over the three coding times.

5.4. RESULTS: FIT BETWEEN INDIVIDUAL PILOT AND MODELED PILOT

PERFORMANCE

5.4.1. Match Between Individual Pilot and Modeled Pilot Plans

5.4.1.1. Mean percent plan match. Using the computation method previouslydescribed, we obtained the mean percent matches between the individual pilotsand models as a function of expertise. The mean percent matches on the cognitiveplans were 89%, 88%, and 88% for novices, intermediates, and experts respectively.The same matches on the action plans were 80%, 79%, and 78% for the threeexpertise groups respectively. Given that we have modeled 25 pilots over 7 segmentsof £ight using a very small window of data from a pilot, we obtained high matchesbetween the pilots and models. The average ¢t between the pilots and modelswas greater for the cognitive plans than for the action plans. This is essentiallyan artifact of the superordinate nature of cognitive plans compared to action plans.For example, many different combinations of the monitor-display action plans (e.g.the altimeter, the attitude indicator, and VSI) could be executed in the processof accomplishing one monitor-status cognitive plan (e.g. monitor altitude).

Overall the models showed a high ability to predict how the pilots would perform.We use the term `predict' because the knowledge base used in each simulation wasbased on a subset of the pilot performance data. Once simulations began, the models£ew the £ight segments in an automated fashion by ADAPT. It suggests thatADAPT, a comprehension-based model of adaptive planning is e¡ective for under-standing the constraint-based activation of pilot knowledge in £ight.

5.4.1.2. Effects of expertise, task complexity, and maneuvering stage. To determineif the ¢ts vary as a function of expertise, task complexity, and maneuvering stage,ANOVAs were conducted on the percent matches of the cognitive and action plansseparately with expertise (novice, intermediate, and expert) as between-participant variable, and task complexity (single-, double- and triple-task) andmaneuvering stage (initiating, maintaining and ¢nishing) as within-participantvariables. The analyses resulted in null effect of expertise and task complexityon percent matches. But there is a main effect of maneuvering stage on the matchesfor the cognitive plans, F(2, 44) � 4.26, MSE = 0.013, p < .02, and for the actionplans, F(2, 44) � 29.30, MSE � 0.010, p < 0:1. Table 6 shows the mean percentmatches as a function of expertise and maneuvering stage. The ¢t was lower atthe maintaining stage than at the other stages for all the expertise groups. This


is due to the greater variability of plans at the maintaining stage. The selection ofplans (actions) are more constrained by the goal(s) of the segment when the model(the pilot) establishes a control change at the initiating stage and releases the controlchange (e.g. roll out and level off) at the ¢nishing stage because the plans to performthose control changes are associated with the goal(s). However, the maintainingstage is likely to involve all sorts of monitoring rather than the goal-constrainedcontrol changes. The less constraints could allow the greater variability of planselection at the maintaining stage.

5.5. SUMMARY OF RESULTS

ADAPT matches individual pilots' performance quite well. Using each pilot'sknowledge base created from very small windows of performance, the modelwas effective in predicting what information pilots will attend to as a functionof expertise and £ight situation (see Table 6). Given that we have modeled 25 pilotsover 7 segments of £ight, and used such rigorous model validation procedures, thisis a signi¢cant ¢nding. It suggests that ADAPT, a constraint-based model of adapt-ive planning is effective for understanding the constraint-based activation of pilotknowledge in £ight.

6. General Discussion

We have shown how a comprehension-based model accounts for a signi¢cantamount of pilot visual attention and £ight performance. Using a model basedon the construction-integration theory of comprehension, we developed individualknowledge bases using a small subset of pilot performance data. The trained knowl-edge bases were used by ADAPT to `£y' £ight segments £own by individual pilots,and this allowed us to predict signi¢cant aspects of user performance.

Table VI. Mean percent match on cognitive and action plans between models and pilots as a func-tion of expertise and maneuvering status, M � mean, SD � standard deviation of the mean

Novice Intermediate ExpertStatus M SD M SD M SD

Cognitive PlansInitiating 89 18 88 10 94 5Maintaining 87 7 85 14 82 14Finishing 89 9 89 8 87 7

Action PlansInitiating 85 14 82 10 87 9Maintaining 74 13 72 15 68 12Finishing 81 10 81 9 78 12


Methodologically, what we have done is applied rigorous training and testingmethods more commonly used by researchers in machine learning to develop apredictive and descriptive knowledge-based model. Modeling twenty-¢ve individualpilots £ying seven segments of £ight allows us to rigorously test the qualitativeand quantitative ¢t of the model to the actual performance data. That is, wecan go beyond mere speculation that the model provides a `good description' ofthe data. ADAPT predicts many aspects of pilot performance during £ight.

The present work does not focus on di¡erentiating the ADAPT architecture fromthat used by ACT-R and Soar, two major models of cognition. The threearchitectures share many attributes including the use of declarative and proceduralknowledge. What distinguishes the three models is how the role of problem solvingcontext is represented, and how it in£uences knowledge activation and use. InSOAR, episodic knowledge is used to represent actions, objects and events thatare represented in the modeled agent's memory (e.g. Rosenbloom, Laird & Newell,1991). This knowledge in£uences the use of procedural and declarative knowledgeby impacting the activation of knowledge based on the context of historical use.In ACT, the analogical processes used to map similarities between problem-solvingsituations simulates the interpretive use of knowledge in a new context.

In the present model, context is not represented as historical memory or governedby an analogical process. Rather, the in£uence of context is to constrain the spread ofknowledge activation based on the con¢gural properties of the current task situationusing low-level associations. Certainly this results in a model that covers a moremodest range of cognitive behaviors than those examined by SOAR and ACT-Rresearchers (e.g. VanLehn, 1991). However, the present rigorous test of predictivevalidity suggests that this simplistic approach to understanding adaptive planninghas signi¢cant promise.

An important strength of the present model is that it has been applied to such awide variety of cognitive phenomena using very few assumptions and very little par-ameter ¢tting. One weakness is that greater parsimony in terms of assumptions andparameter ¢tting has lead to less than perfect model ¢ts to the human data. Thereis clearly a tradeo¡ between parameter ¢tting and parsimony. In this case, a rela-tively parsimonious model has provided reasonable ¢ts to highly complex humanperformance in a dynamically changing environment. In addition, this workhighlights the importance and necessity of turning toward the building of predictiveindividual models of human performance, accounting for di¡erences at the partici-pant level, rather than simply describing for aggregate performance.

There exists today an enormous amount of interest in adaptive training systems, acomponent of which is the model of student knowledge or `student model.' The pre-sent ¢ndings have clear implications for the acquisition and adaptation of studentmodel components of intelligent instructional systems. Our results suggest that asigni¢cant portion of pilot visual attention and control movements can be antici-pated during simulated £ight. Given this, then the anticipated pilot behavior couldbe used to customize instruction, or even to warn pilots about upcoming situations


in an e¡ort to modify their behavior. Current research in our lab is focussed on suchinstructional e¡orts.

Appendix

EQUATIONS FOR CONSTRUCTION AND INTEGRATION

1. Construction

At the construction process, the following equations de¢ne the strengths of relation-ships between propositions represented in the knowledge base.

(A) Argument overlapThe strengths (Wij;Wji) of relationships for Pi;Pj 2 fworld knowledge, generalknowledgeg are de¢ned as follows:

For i 6� j;

Wij �Wji �Woverlap � �numbers of arguments overlapped�;where Woverlap � 0:4:For i � j;

Wij � 1:0

The strengths (Wik;Wki) of relationships for the same set of Pi and Lk 2 {planknowledge} are de¢ned as follows:

Wik �Wki �Woverlap � �number of arguments overlapped�;where Woverlap � 0:4:

(B) Plan-world inhibitionIf outcomes of Lk exist in-the-world knowledge, the strength Wki of an asymmetricrelationship for Lk 2 {plan knowledge} from Pi 2 {world knowledge} is de¢nedas follows:

Wki �Winhibition; where Winhibition � ÿ10:0

If Pj, an outcome of Lk, with the trace predicate exists in-the-world knowledge, thestrength Wkj of an asymmetric relationship for Lk 2 {plan knowledge} fromPJ 2 {world knowledge} is de¢ned as follows:

Wkj �Woverlap � �number of arguments overlapped�; where Woverlap � ÿ0:4

(C) Interplan relationships


Asymmetric causal relationships for Lk;Ll 2 {plan knowledge} are de¢ned asfollows:

For k 6� 1;if Lk supports Ll;Wkl �Wexcitation; where Wexcitation � 0:7if Lk inhibits Ll;Wkl �Winhibition; where Winhibition � ÿ10:0For k � l;

Wkl � 1:0

2. Integration

A set of nodes interconnected by the construction process is represented by:

�X1; . . . ;Xi; . . .XM;Y1; . . . ;Yj; . . .YN�

where Xis represent world knowledge and Yjs represent general and plan knowledge,and M and N are the number of nodes representing the respective knowledge. Thepattern of activation after n-th iteration can be expressed by a vector,

�A�n� � �AX1 ; . . . ;AXi ; . . . ;AXM ;AY1 ; . . . ;AYj ; . . . ;AYN �

where AXi and AYj represent activation values of world knowledge and the otherknowledge respectively, and the strengths in the constructed network by a matrix,C.

The initial activation is set as follows:

A0Xi� 1:0 �1W iWM�

A0Yj� 0:0 �1W jWN�

The activation vector after n-th iteration, �A�n� is de¢ned as follows:

A�n�Xi� 1:0

A�n�Yj�

max�0:0;A�n�YjPNk�1 max 0:0;A�n�Yk

� �where unnormalized activation vector calculated by matrix multiplication,

�A�n� � C � �A�nÿ1�

is normalized.When average change of the activation vector is below 0.0001, that is,

1n�XN

j�1��A�n�j ÿ A�nÿ1�j

�� < 0:0001

the network is considered to be stabilized. The activation vector, �A�n� becomes the¢nal activation vector.


Acknowledgements

This research was supported by funding from the Of¢ce of Naval Research. Theauthors wish to thank Drs Susan Chipman, Harold Hawkins, and Terry Allerd fromONR for their support of this effort. We would also like to thank Drs GaryBradshaw, Alfred Kobsa, and three anonymous reviewers for their comments onprevious versions of this manuscript.

References

Allen, J. F. and Perrault, C. R.: 1980, Analyzing intention in utterances.Arti¢cial Intelligence,15, 143^178.

Anderson, D. A.: 1989, Arti¢cial Intelligence and Intelligent Systems. Wiley and Sons, NewYork.

Anderson, J. R.: 1993, Rules of the Mind, Erlbaum, Hillsdale, NJ.Anderson, J. R. and Douglass, S.: (in press). Visual attention and problem solving. Journal of

Experimental Psychology: Learning, Memory and Cognition.Broadbent, D. E.: 1993, Comparison with human experiments. In: D. E. Broadbent (ed.), The

simulation of human intelligence. Blackwell, Cambridge, MA, pp. 198^217.Chase, W. G. and Ericsson, K. A.: 1982, Skill and working memory. Psychology of Learning

and Motivation, 16, 1^58.Doane, S. M., Mannes, S. M., Kintsch, W. and Polson, P. G.: 1992, Modeling User Command

Production: A Comprehension-Based Approach. User Modeling and User AdaptedInteraction, 2(3), 249^285.

Doane, S. M., McNamara, D. S., Kintsch, W., Polson, P. G. and Clawson, D.: 1992, Promptcomprehension in UNIX command production.Memory and Cognition, 20(4), 327^343.

Doane, S. M. and Sohn, Y.W.: 1999, Instrument Scanning as a Function of Piloting Expertise.Manuscript submitted for publication.

Doane, S. M., Sohn, Y. W., Adams, D. and McNamara, D. S.: 1994, Learning frominstruction: A comprehension-based approach. Proceedings of the 16th Annual Conferenceof the Cognitive Science Society, Atlanta, GA: Erlbaum, pp. 254^259.

Doane, S. M., Sohn, Y. W., McNamara, D. S. and Adams, D.: (in press). Comprehension-based skill acquisition. Cognitive Science.

Dogan, P.: 1991, The instrument £ight training manual. Santa Clarita, CA: Aviation BookCompany.

Dreyfus, H. L.: 1996, Response to my critics. Arti¢cial Intelligence, 80, 171^191.Dreyfus, H. L.: 1992,What computers still can't do: A critique of arti¢cial reason. Cambridge,

MA, MIT Press.Ericsson, K. A. and Kintsch, W.: 1995, Long-term working memory. Psychological Review,

102, 211^245.Federal Aviation Administration: 1980, Instrument £ying handbook. U.S. Department of

transportation.Fox, J., Merwin, D., Marsh, R., Sohn, Y., Doane, S., Kramer, A., Lintern, G. and Wickens,

C.: 1995, Modeling pilot skill acquisition. Paper presented at the Biennial Aviation Psy-chology Conference. Columbus, OH.

Gernsbacher, M. A.: 1990, Language comprehension as structure building. Erlbaum, Hillsdale,NJ.


Hoecker, D. G., Corker, K. M., Roth, E. M., Lipner, M. H. and Bunzo, M. S.: 1994, Man-machine design and analysis system (MIDAS) applied to a computer-basedprocedure-aiding system. Proceedings of the 38th Annual Meeting of the Human Factorsand Ergonomics Society, Part 1, Human Factors and Ergonomics Society, Nashville,TN, pp. 195^199.

Holyoak, K. J.: 1991, Symbolic connectionism: toward third-generation theories of expertise.In: K. A. Ericsson and J. Smith (eds.), Toward a general theory of expertise, CambridgeUniversity Press, pp. 301^336.

Holyoak, K. J. and Thagard, P.: 1989, Analogical mapping by constraint satisfaction.Cognitive Science, 13, 295^355.

Just, M. A. and Carpenter, P. A.: 1992, A capacity theory of comprehension. PsychologicalReview, 99, 122^149.

Kintsch, W.: 1998, Comprehension: A Paradigm for Cognition, Cambridge University Press,Cambridge, MA.

Kintsch, W.: 1988, The use of knowledge in discourse processing: A construction-integrationmodel. Psychological Review, 95, 163^182.

Kintsch, W.: 1994, The psychology of discourse processing. In: M. A. Gernsbacher (ed.),Handbook of Psycholinguistics, Academic Press, San Diego, pp. 721^739.

Kitajima, M. and Polson, P. G.: 1995, A comprehension-based model of correct performanceand errors in skilled, display-based, human-computer interaction. International Journalof Human-Computer Studies, 43, 65^99.

Lovett, M. C. and Anderson, J. R.: 1996, History of success and current context in problemsolving. Cognitive Psychology, 31, 168^217.

Mannes, S. M. and Doane, S. M.: 1991, A hybrid model of script generation: Or getting thebest of both worlds. Connection Science, 3(1), 61^87.

Mannes, S. M. and Kintsch, W.: 1991, Routine computing tasks; Planning as understanding.Cognitive Science, 15(3), 305^342.

Meyer, D. E. and Kieras, D. E.: 1997, A computational theory of executive cognitive processesand multiple-task performance: Part 1. Basic Mechanisms. Psychological Review, 104,3^65.

Miller, J. R. and Kintsch, W.: 1980, Readability and recall of short prose passages: A theor-etical analysis. Journal of Experimental Psychology: Human Learning and Memory, 6,335^354.

Newell, A.: 1987, Uni¢ed theories of cognition (The 1987 William James Lectures). HarvardUniversity Press, Cambridge, MA.

Recker, M. M. and Pirolli, P.: 1995, Modeling individual di¡erences in students' learningstrategies. The Journal of the Learning Sciences, 4, 1^38.

Rosenbloom, P. S., Laird, J. E. and Newell, A.: 1991, Toward the knowledge level in SOAR:The role of architecture in the use of knowledge. In: K. VanLehn (ed.), ArchitecturesFor Intelligence. Erlbaum, Hillsdale, NJ, pp. 75^112.

Rosenbloom, P. S., Laird, J. E., Newell, A. and McCarl, R.: 1991, A preliminary analysis ofthe SOAR architecture as a basis for general intelligence. Arti¢cial Intelligence, 47,289^325.

Rouse, W. B., Geddes, N. D. and Curry, R. E.: 1988, An architecture for intelligent interfaces:Outline of an approach to supporting operators of complex systems. Human-ComputerInteraction, 3, 87^122.

Schank, R. C. and Abelson, R. P.: 1977, Scripts, Plans, Goals and Understanding. Erlbaum,Hillsdale, NJ.


Schmalhofer, F. and Tschaitschian, B.: 1993, The acquisition of a procedure schema from textand experiences. Proceedings of the 15th Annual Conference of the Cognitive ScienceSociety. (pp. 883-888). Erlbaum, Hillsdale, NJ, pp. 883^888.

Selfridge, O. G.: 1998, Pandemonium: A paradigm for learning. In: J. A. Anderson and E.Rosenfeld (eds.), Neurocomputing: Foundations of Research. MIT Press, Cambridge,MA, pp. 117^122

Sohn, Y. W. and Doane, S. M.: 1997, Cognitive constraints on computer problem solvingskills. Journal of Experimental Psychology: Applied, 3, 288^312.

Thagard, P.: 1989, Explanatory coherence. Brain and Behavioral Sciences, 12, 435^467.Turner, A. A. andGreene, E.: 1978, Construction and use of a propositional text base.Catalog

of Selected Documents in Psychology, 8 MS, 1713^1758.van Dijk, T. A. and Kintsch, W.: 1983, Strategies of discourse comprehension. Academic

Press, New York.VanLehn, K.: 1988, Student modeling. In: M. C. Polson and J. J. Richardson (eds.), Foun-

dations of intelligent tutoring systems. Erlbaum, Hillsdale, NJ, pp. 55^76.VanLehn, K., (ed.): 1991, Architectures For Intelligence. Erlbaum, Hillsdale, NJ, 1991,

pp. 75^112.Wilensky, R.: Why John married Mary: Understanding stories involving recurring goals.

Cognitive Science, 2, 235^266.Zachary, W.: 1996, Interface agents in complex systems. In: C. A. Ntuen and E. H. Park (eds.),

Human Interaction with Complex Systems: Conceptual Principles and Design Practice.Kluwer, Norwell, MA, pp. 397^401.

Authors' Vitae

Dr. Stephanie M. Doane, Department of Psychology, P.O. Box 6161, MississippiState University, Mississippi State, MS 39762, USA. Dr. Doane received her B.A.in Experimental Psychology from the University of California Santa Barbara,her M.S. in Experimental Psychology from Villanova University, her Ph.D. inCognitive Psychology from the University of California Santa Barbara, and com-pleted her Post-Doctoral Training at the University of Colorado Institute ofCognitive Science. She is an Associate Professor of Psychology at Mississippi StateUniversity. Her research interests include strategic aspects of skill acquisition,expertise, and computational models of user cognition.

Dr. Young Woo Sohn, University of Connecticut, Department of Psychology, 406Babbidge Road, U-20, Storrs, CT, USA. Dr. Sohn received his B.A. degree in Busi-ness Administration from Korea University in 1986, his M.A. in Sociology fromUniversity of Missouri at Columbia in 1989, and his Ph.D. in Psychology from Uni-versity of Illinois at Urbana-Champaign in 1999. He is now Assistant Professor ofPsychology at University of Connecticut. His research interests lie in expertiseand skill acquisition, perceptual learning, and cognitive modeling.
