cognitive task analysis of expertise in air traffic control

29
This article was downloaded by: [The University Of Melbourne Libraries] On: 13 September 2014, At: 04:30 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK The International Journal of Aviation Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hiap20 Cognitive Task Analysis of Expertise in Air Traffic Control Thomas L. Seamster , Richard E. Redding , John R. Cannon , Joan M. Ryder & Janine A. Purcell Published online: 13 Nov 2009. To cite this article: Thomas L. Seamster , Richard E. Redding , John R. Cannon , Joan M. Ryder & Janine A. Purcell (1993) Cognitive Task Analysis of Expertise in Air Traffic Control, The International Journal of Aviation Psychology, 3:4, 257-283, DOI: 10.1207/s15327108ijap0304_2 To link to this article: http://dx.doi.org/10.1207/s15327108ijap0304_2 PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and

Upload: janine-a

Post on 06-Feb-2017

215 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Cognitive Task Analysis of Expertise in Air Traffic Control

This article was downloaded by: [The University Of MelbourneLibraries]On: 13 September 2014, At: 04:30Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number:1072954 Registered office: Mortimer House, 37-41 Mortimer Street,London W1T 3JH, UK

The International Journalof Aviation PsychologyPublication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/hiap20

Cognitive Task Analysisof Expertise in Air TrafficControlThomas L. Seamster , Richard E. Redding ,John R. Cannon , Joan M. Ryder & Janine A.PurcellPublished online: 13 Nov 2009.

To cite this article: Thomas L. Seamster , Richard E. Redding , John R.Cannon , Joan M. Ryder & Janine A. Purcell (1993) Cognitive Task Analysisof Expertise in Air Traffic Control, The International Journal of AviationPsychology, 3:4, 257-283, DOI: 10.1207/s15327108ijap0304_2

To link to this article: http://dx.doi.org/10.1207/s15327108ijap0304_2

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of allthe information (the “Content”) contained in the publications on ourplatform. However, Taylor & Francis, our agents, and our licensorsmake no representations or warranties whatsoever as to the accuracy,completeness, or suitability for any purpose of the Content. Anyopinions and views expressed in this publication are the opinions andviews of the authors, and are not the views of or endorsed by Taylor& Francis. The accuracy of the Content should not be relied upon and

Page 2: Cognitive Task Analysis of Expertise in Air Traffic Control

should be independently verified with primary sources of information.Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilitieswhatsoever or howsoever caused arising directly or indirectly inconnection with, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private studypurposes. Any substantial or systematic reproduction, redistribution,reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of accessand use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 3: Cognitive Task Analysis of Expertise in Air Traffic Control

THE ~ P U I O N A L JOURNAL OF AVPXION PSYCHOLOGY, 34). 257-283 Copyight 0 1993, LPwace Edb.ttm W g Inc

Cognitive Task Analysis of Expertise in Air Traffic Control

Thomas L. Seamster Carlow Internatwnal

FalLs Church, Virginia

Richard E. Redding and John R. Cannon Human Technology McLean, Virginia

Joan M. Ryder and Janine A. Purcell CHI Systems

Spring House, Pe~sylvania

A cognitive task analysis was performed to analyze knowledge structures, mental models, skills, an:! strategies of en route coatrollers to provide an understanding of the key cognitive components of the air traf£ic controller's job. This article presents the resu:ts cf three procedures as they contributed to an understanding of controller expertise: paper -mblcm solving, per fomma modeling, and structured problem solving. The procedures resulted in the identification of (a) 13 primary tasks, @) a mental mcdel representing expert controller's organization of domain knowledge, (c) h e categories of controller strakgies, and (d) a hierarchy of goals. T h a e results are being used to specify the instructional content and sequencing for the new Federal Aviation Administration en route air traffic control curriculum.

A cognitive task analysis (CXA) was conducted as part of the Federal Aviation Administration (FAA) redesign of its en route air traffic control (ATC) curriculum. This is one of the first uses of a CT'A to support development of an entire curriculum

Requests for reprints should be sent to Richard E Redding, 806 W-n Drive NE, Apanment 304, Leesburg, VA 22075.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 4: Cognitive Task Analysis of Expertise in Air Traffic Control

redesign for the training of a complex high-performance task in civil aviation. This two-phase effort analyzed controller knowledge structures, mental models, skills, and strategies. The first phase included the development of a preliminary mental model with its associated tasks and cognitive strategies. In the second phase, the model was validated, and the tasks and strategies were extended to include a greater portion of the en mute controilcr's job.

This CZA was used to expand the new en mute ATC curriculum to include the cognitive skills of planning, decision making, and workload management. The goals of the new curriculum are to provide training in the least amount of time required to develop the necessary knowledge and skills, and to motivate the trainees to strive for the highest level of professional competence. The curriculum redesign project will ultimately span the entire framework of training theory (see Cannon- Bowers, Tamenbaum, Salas, & Converse, 1991), from "What should be trained?" to "Is the training effective?" The present CXA addresses the first question, which deals with the identification of the knowledge, skills, and strategies that should be taught in ATC. The training implications of the CTA results will link the character- istics of controller expertise to the controller training process.

This effort demonstrates an application of tXA to complex, time-constrained, multitasking environments that involve a high degree of cognitive processing on the part of the system operator. These complex work envimnments include aircraft flight decks, combat information centers, hospital operating rooms, and ATC centers. These envimnments share a number of characteristics: time-constrained multiple tasks; a complex, dynamic information environment; and teams of oper- ators that need to coordinate in order to perform tasks. As such, they require a special form of expertise involving not only extensive domain knowledge, but also efficient problem-solving strategies that can be implemented within the time-crit- ical limits of the task. This contrasts with the more traditional research on expertise in such domains as basic physics skills (Chi, Feltovich, & Glaser, 1981), chess (de Groot, 1965), Go (Reitman, 1976), and computer programming (Adelson, 1981), which involve primarily sequential, deliberate reasoning rather than task prioritiza- tion and workload management.

Traditional analyses of expertise have addressed how experts represent domain- specific information (Adelson, 1981). These studies of expertise grew out of de Groot's (1965) classic study of master chess players, in which he attributed their superior memory for midgame board pieces to their ability to classify or cluster groups of pieces based on their positions in specific board configurations. Subse- quent studies of expertise emphasized the way experts organized or clustered information in human memory. Chi, Fcltovich, and Glaser (1981) provided a new perspective by looking not only at the structure of the mental representation, but also at how advanced physics students actually used that expert representation. They concluded that experts at solving physics problems form schemas based on the principles implied in the problem. These scbemas contain information on how and when to use the relevant principles.

In time-constrained tasks, for which timing is a crucial aspect of expert perfor- mance, the emphasis on the "how" and "when" to use a particular strategy is even

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 5: Cognitive Task Analysis of Expertise in Air Traffic Control

greater. Clearly, this time-critical, complex form of expertise requires more than just having the appropriate knowledge structures; those knowledge structures must be organized in a highly eficienl manner to optimize retrieval.

The CTA discussed in this article illuminates several issues in the application of cognitive theory to training methodology. It emphasizes the importance of mental models in redesigning training curricula. It also provides a set of preliminary characteristics common to expertise in time-constrained, complex tasks, a form of expertise required In a number of high-performance jobs. This CTA represents a transition from emphasis on knowledge structures lo emphasis on the dynamic aspects of mental models (Redding & Seamster, in press). This a p p c h analyzes not only how experts represent domain knowledge, but also when and how they use that knowledge in response to time-critical task demands. Conceptual and procedural knowledge is necessary for the performance of these tasks, but the framework that permils rapid encoding and decoding of sector data appears to be the critical requirement. This deductive framework allows the expert controller to formulate an eflicient mental model that is a dynamic and computational represen- tation (Wilson & Rutherford, 1989) of ATC-critical events.

The present CTA was based on a modification of the integrated task-analysis methodology (Redding, 1993#3, Ryder & Redding, 1993) involving several itera- tions of data collection, analysis, and interpretation. Phase I of this CTAwas a broad effort that included seven data collection techniques: (a) structured and un- structured interviews, @) critical-incidents interviews, (c) paired paper problem solving, (d) cognitive-style assessment, (e) simulated performance modeling, and (f) structured problem solving. Phase I resulted in a preliminary mental model for the en route controller; specification of the primary controller tasks and their subgoals; and the identification of controller strategies, goals, and methods.

Phase I1 of the CTAextended and validated the key results, including the mental model, tasks and related subgoals, and expert strategies. In addition, Phase I1 analyzed the critical cues associated with work overload. For this phase, the original data were subjected to further analysis, andadditional overload data were collected and analyzed. In Phase 11, there was a shift b m the broader cognitive concerns to a focus on the mental model and strategies as they relate to controller tasks. This iterative process led from the analysis of general controller expertise to a narrower analysis of conmller expertise under conditions of heavy workload.

To place the present CI'Aeffort into perspective, it is useful to briefly compare i t with another recent controller task analysis. CTA Incorporated (1990) analyzed operational tasks as the en route controller interacted with the system. The analysis began with the identification of system events requiring controller response. The controller responses were decomposed into three levels: activity level, sub-activity level, and task level. The task was a specific statement of a single job action consistent with the controller's inputs to the system and with outputs from the system. The emphasis was on a detailed characterization of each task, specifying information requirements, frequency, and task cri ticality from a system functional- ity perspective.

By contrast, the present CTA emphasized the analysis of those controller tasks

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 6: Cognitive Task Analysis of Expertise in Air Traffic Control

that are units of goal-directedbehavmr under the air W~ccontroller's general goal of providing safe, orderly, andexpeditious fbw of air traffic. The tasks are analyzed ftom a cognitive perspective identifying the task subgods and the trigges that activate the individual task. The detailed analysis focuses on the task triggers and the messages passed to and from the mental model during subgoal execution. This CXAemphasizes the relation of tasb to the expert controller's mental model with a focus on planmg and maintaining situation awareness.

The rest of this article ptestnts the mthods, results, and discussion of three of the seven CTA analyses. F i the paired paper problem-solving analysis is pre- sented, with its results showing controller goak aami different levels of experi- ence. Second, the mental-model development and task decomposition are reported with their resulting tasks and their interaction with the expert controller's mental model. Last, the stmctur&problern-solving analysis is presented with the resulting controller strategy usage by novice, intermediate, and expert en route air traffic controllers. (For a complete description of the entire CIA, see Redding et al., 1990, Redding, Ryder, Scamster, Purcell, & Cannon, 1991.)

ANALYSIS 1 : PAIRED PAPER PROBLEM SOLVING

This analysis compared how controllers with different levels of expertise develop and solve static air traffic problems. Participants werr: grouped into pairs, which resulted in a form of collaboration that emmuaged the pair of controllers to explore a richer set of problem solutions than would be elaborated by a single participant.

Participants. The 18 participants (all male) fell into three group, each with a different level of AI'C experience: 7 experts, 5 intermediates, and 6 novices. The experts includedbothsupervisors andFull Perfomrance Level (FPL)um&ollers with more than 5 years of FPL experience. The intenncdiates all had less than 1 year of FPLexperience, and the novices wcn developaxntak still at the training stage.

Procedures. Paired problem solving (Means & Gott, 1988) involves part~c- ipants developing problems individually and thtn working in pairs presenting them to one another. This technique allowed the reasoned development of solution sequences acms three levels of controlkr expertise. These problems were static, and the controllers could modify their solution sequences by discussing their reasons for doing so. This technique thus complemented Ume used in Analyses 2 and 3, where problems were presented through real-time simulation, thus providing both concurrent and retrospective protocols.

Participants worked within their experience group; experts were paired only with other experts, intermediates with intermediates, and novices with novices. There were two sessions. In the first session, participants were asked to develop a difficult controller problem on paper. Each participant was given a map and a set

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 7: Cognitive Task Analysis of Expertise in Air Traffic Control

of blank flight strips. They were instructed to mark the positions of aircraft on the map at the time of the problem. After the problem was developed, the participant was then asked to generate the optimum solution by identifying each action, the goal motivating that action, the consequence of the action, and the information required to solve the problem.

During the second session, the participant who generated the problem presented it to another member of the same group, who was asked to explain his solution. The individual solving the problem was asked to talk through his solution sequence. (For a more detailed description of the methods, see Redding et al., 1990.)

Analysis. The taskgal was defined as identifying the least costly means of resolving an undesirable controller situation. The transcriptions were analyzed to determine the individual goals for each step in the problem-solving sequence. These goals, based on the definition of the Goals, Operatiros, Methods, and Selection Rules model (Car4 Moran, & Newell, 1983), are the desired end states. It was assumed that the controller's overall goal is to resolve undesirable situations. Therefore, the goals were classified according to what the controller thinks would be likely to happen if no action were taken.

Results

Controller goals were categoxized and prioritized. The resulting categories are related to controller consequences as opposed to aircraft consequem. In order of descending priority, the key controller goals are to avoid (a) violation of minimum separation standards, @) deviations from standard operating procedures, (c) disorder that may result in cognitive work overload, and (d) making unnecessary requests of the pilot.

Figure 1 shows the key goals reported by novices at each of the first 10 steps in problem solving. The percentages represent the means for six novice problems. Novices initially tended to concentrate on resolution of potential violations. As these violations were resolved, they gradually increased their concentration on potential deviations. During the first 5 steps, novices dealt exclusively with violations and deviations. n e i r return to violation goals in Steps 9 and 10 may indicate that they did not identify all violation situations at the beginning of the problem solution. It is likely that they discovered additional situations as they further studied the problem during the exercise.

Figure 2 depicts the distribution of expert controller goals. These percentages are the means for seven expert problems. In comparing Figure 1 with Figure 2, several differences are apparent. The experts did not focus as heavily on violation goals throughout the problem as did novices. Rather, they reported goals in a recursive manner, addressing violations and deviations alternately as they pro- gressed through the solution steps. Experts appeared to be more global in their analysis of problem solutions because they not only responded to threatening situations, but they also addressed goals that would improve sector order or meet pilot requests. Therefore, expert controllers appeared to have a more comprehens- ive view of the evolving air traffic situation.

Experts appeared to take an iterative approach to dealing with violations, in that

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 8: Cognitive Task Analysis of Expertise in Air Traffic Control

- 50 ................................................................. - a l 40 ................................................................. g B .................. 0 30 ......................................... +.- +.

20 ............................. ' 0

o a- a-m 1 2 3 4 5 6 7 8 9 1 0

Soqumo Stop Numbor

-Violations

Deviations

--Disorder

FIGURE 1 Percentage of noviots' responses in each goal category for the first 10 steps in problem solving.

Exports' Goal Catogorlos

V i o l a t i o n s

' - ' Deviations

-X- Dimorder

Soquonco Slop Wurnbn

FIGURE 2 Percentage of exputs' responsg in eacb goal category for the fiat 10 steps in problem solving.

they shifted their attention to other goals (e.g., dealing with deviations) during less critical points in the problem. By contrast, novice controllers dealt sequentially and continually with violations, with a substantial increase in their attention to &via- tions only in the later s t e p of Ihe problem. Overall, novices concenlrated their attention on violations that may be cognitively more demanding for nwices, and this may have reduced their ability to attend to other goals.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 9: Cognitive Task Analysis of Expertise in Air Traffic Control

ANALYSIS 2: MENTAL MODEL AND TASK DECOMPOSITION

This analysis was designed to model the prioritization &cision making of expert en route controllers, using the Cognitive Networks of Tasks (COGNET) framework (Zachary, Ryder, Ross, & Weiland, 1992). COGNET models real-time multitasking environments as a set of tasks, each of which represents a goal-directed procedure for resolving some aspect of the overall situation. The tasks are coordinated-they all use a common problem representation or mental model. The flow of cognitive processing in the COGNET model resides at any moment in a specific task. The controller's attention shifts from one task to another based on the current situation, as represented in the mental model. The mental model is continually updated as the result of task performance or external events, which are registered through percep tual processes.

Tasks in the COGNET model are described using a modified GOMS notation (Card et al., 1983). Each task has triggering conditions (which state when a task should be initiated) and a set of behavioral and cognitive subgoals comprising the task. The mental model is represented as a blackboard structure (see Nii, 1986) with separate panels for major categories of information and levels in each panel indicating the organization of information within the category.

The objective of this analysis was to derive the structure and content of the mental model of en route N C , to decompose the job into tasks, and to model each task. During Phase I, a preliminary set of tasks and a mental model were defined. During Phase 11, the model was extended and validated. Extension of the model involved further analysis of the original data to provide greater detail for the task subgoals, task triggers, and the contents of the mental model. In addition, a validation study was conducted to determine whether the mental mo&l and task decomposition provided a useful irdmework for describing controller performance.

Method

Participants. For the original data collection, 5 supervisor and/or FPL con- trollers were used. Their mean age was 47.2 years (SD = 3.27 years), and they had a mean of 18.8 years of FPL experience (SD = 3.27 years). During Phase 11, 2 additional FPLs and 4 FAA instructors acted as participants and subject matter experts (SMEs). All participants were male.

Procedures. During the original data collection, each participant controlled a simulated air traffic sector and worked four different problems. These problems were presented on an ATC simulator with the capability to simulate radar displays, pilot communication, and communication to other sectors. In these problems, the participant acted as a single controller working the entire sector. This situation is not uncommon during periods of light to moderate air traffic. l b o of the problem scenarios presented 65% task complexity. This level of traffic volume and problem difficulty is generally lower than a real-time working situation. Complexity is an FAA rating of workload based on factors such as traffic volume, weather, and

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 10: Cognitive Task Analysis of Expertise in Air Traffic Control

emergencies. I he remaining two problems presented 1004b complexity, or a full workload. Individual performance was videotaped, and participants were allowed to work each of the four problem scenarios uninterrupted for 20 to 35 min.

A week after the problem scenario session, individual participants were asked Lo review their poblem solutions while the scenario was replayed; they were asked to describe each action taken, their intentions, and expecled outcomes. They were prompled for a full description bough the use of the following probes: (a) "What were you paying attention to here?" @) "What's going on now that's most important?" (c) "What were you doing?" (d) "Why?" (e) "What was your goal'?" ( f ) "What activity is it part of?" (g) "How does it relate lo what you did just before it?" and (h) "How did you know what needed to be done next?"

In Phase 11, the model was submitted to an exlension and validatio~l process. The model extension involved further analysis of the original data to provide greater detail to the task subgoals, task triggers, and the contents of the mental model. in addition, 2 SMEs reviewed a subset of the original videotapes, providing alternative methods and solutions for those problems in order to account for different strategies and methods within the model. The SMEs then reviewed and evaluated the extended mental model.

After the model was completed, a validation study was conducted to assess the construct validity of the mental model and task decomposition (i.e., the extent to which the model actually reflects controllers' knowledge organization, action sequences, and explanations of their goals and intentions in performing actions). For the validation study, data for solving a work overload problem were collected from 2 experienced FPLs. It was hypothesized that controllers would be more likely to demonstrate their expertise when working under a heavy workload. As in the procedure described earlier, the participants were allowed, to work through the entire problem, and then during playback they were prompted to discuss their controllcr actions and thought processes.

Analysis. Model construction was a three-part process that included task decomposition, detailed task modeling, and problem representation of the mental- model development (see Figure 3). Task decomposition was the first step. It involved analyzing the videotaped problems and their associated protocols to determine segments of related activity and the goals these activity segments were intended to accomplish. Each segment of related activity in service of a goal was considered a task for that participant. The tasks identificd for each participant were then merged across the group of experts, forming a common task list. This list of common tasks was subjected to a refinement process in which each task was compared with the others Lo determine whether (a) task A was a kind of task B, (b) task A was a part of task B, and (c) tasks A and B were instances of some more abstract task C. This process resulted in a preliminary task list.

The second and third parts-the detailed modeling of the tasks and the devel- opment of the mental model-were performed in parallel. The subgoals of each task were determined and the general stmturc of the mental model was defined. Observable aspects of the task models (behavioral subgoals) were determined first

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 11: Cognitive Task Analysis of Expertise in Air Traffic Control

A'L'C COGNlTlVE TASK ANA1,YSlS 265

FIGURE 3 Depiction of the model-coastnrc~ion proccss.

because they were explicit in the videotapes of controller actions. The cognitive subgoals were added subsequently because they had lo be derived from the protocols and inferred from the contents of lhe mental modcl (e.g., the traffic typelroule level of the aircraft data panel contains messages in the format [aircraft ID, route type, route]), and cognitive operations were added lo the task modcls to indicate how information in the mental model was updated in the conduct of the tasks. The mental model and lask models were U~cn refined and revised until Uley were consistent and complete. Finally, task triggers were addcd lo the task models, showing what patterns of information in the mental model should trigger that task.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 12: Cognitive Task Analysis of Expertise in Air Traffic Control

This resulted in an integrated model of controller pcrformancc, including task models, conditions for task initiation (lriggcrs), and the associated mcnlal model used by a controllcr to understand the air traffic siluation.

After the model was completed, it was validated using a dilfercnt method of analysis. The data collected were uscd to construct a timeline of task events, or spccific instances of a task. Arcpresentative sample of 46 ksk events was analyzed. Sccnario Umc, controllcr actions (behavioral and cognitive) performed, condition initiating task cvcnl, and information used wcre dctermincd for each lask cvcnt. The task evcnts were then compared lo the 13 primary tasks and the mental modcl to dclcrmine (a) whether the participants actions could be accounted for completely by the task models, (b) whcther the task triggers accurately indicatcd thc conditions for task initiation, and (c) whether the participants described their cognitive processes in lcrms of the panel and levels in the mental model.

Results

The rcsulls of thc task decomposition arc described first, followed by the mental model and the validation results.

Tasks. A rusk was defined as a single, goal-directed activity that would continue to completion if uninterrupted. If the conlroller's job is thought of as a goal hierarchy, then the goal of each task is one level below the conlroller's toplcvel goal of providing sak, orderly, and expeditious air LrafTic flow. The task decomposition emphasizes the cognitive activities thal make up each lask. There- fore, tasks are defined by their goal structure rather than by their behavioral distinctness.

The task decomposition resulted in thc identification of 13 primary conlroller tasks (sec Table 1). The tasks could be composed of behavioral or cognitive subgoals or both. The analysis yielded two tasks that were primarily cognitive: "maintain situation awareness" and "develop and revise sector control plan." Maintain situation awareness is central to all other tasks and is normally undertaken at the beginning of a shift and returned lo whenever possible thereafter. Develop and revise sector control plan is closely couplcd to maintain situation awareness; these two tasks are often done in conjunction with each other because a change in the situation often requires a change in sector plan. As changes in the sector situation lake place, they trigger a specific task to handle the event. For example, when a conflict is noticed while maintaining situation awareness, thal would trigger the "resolve aircraft conflict" task, which would be completed by the controller. If there were no interruptions, the controller's attention would return to maintaining situation awareness.

Task lriggcrs, task subgoals, and subgoal messagcs for each task wcre idcntificd, linking the tasks lo the mental model. Task triggers are represented as and-or statements specifying the conditions under which a task is to be initiated. For example, a trigger to initiate a handoff might be "aircraft less than 30 miles from sector boundary AND no other critical event occurring."

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 13: Cognitive Task Analysis of Expertise in Air Traffic Control

ATC COGNITIVE TASK ANALYSIS 267

TABLE 1 The 13 En Route Air Traffic Control Tasks

Task Definition

Maintain situation awareness Maintain understanding of current and projected positions of aircraft in the sector to determine events that require controller activities

Develop and revise sector control plan Develop and revise a plan for controlling the sector that is current and comprehensive and

Resolve aircraft conflict

Reroute aircraft

Manage arrivals

Manage departures

Manage overflights

Receive handoff Receive pointout

Initiate handoff

Initiate pointout

Issue advisory

Issue safety alert

that handles contingencies Evaluate potential conflictions and implement

means to avoid them Change aircraft routes in response to requests

or situational considerations Establish sequence and routing of aircraft for

arrival into an airport Maintain safe and efficient departure flows

integrated with other sector traffic Maintain safe and efficient overflights and

integration of overflights with other sector traffic

Accept, delay, or deny handoffs Assess and accept or decline or pointout from

another controller Transfer aircraft radar identification and radio

communications to the receiving controller Initiate and complete pointout of aircraft to

the receiving controller Provide information update to a pilot or

another controller Provide mandatory safety warning to a pilot

Once a task is activated, the controller perfonns each of the task subgoals and sub-subgoals until there is an interruption (i.e., a higher priority task is triggered) or until the task is completed. The subgoals may include cognitive operations that use information from the mental model and produce new information that is then added to the mental model (shown hereafter as messages indicating what informa- tion in the mental model is transformed). Within the maintain situation awareness task, for inslance, there are five subgoals (see Redding et al., 1991), with one of them represented as follows:

Subgoal: Evaluate changes in airspace features or procedures . . . WHEN changes occur.

Sub-subgoal: Evaluate/clarify new or changed airspace feature. Message: [Add, modify, or delete <data> on sector airspace panel.] Sub-subgoal: Evaluate/clarify new procedure. Message: [Add, modify, or delete <data> on procedures panel.]

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 14: Cognitive Task Analysis of Expertise in Air Traffic Control

With the task subgoals and sub-subgoals defincd, thc task dcfinition is complctc. The menlal-modcl development look place in parallel wilh this effort, and the results of lhat effort are presented next.

Mental model. The mental modcl of en routc ATC (shown in Figurc 4) is a rcprcscntalion of Ule slalic and dynamic knowledge required to manage a sector. 11s structure implies a conceplual framework used Lo organ i~ ATC knowledge and a slrdlegy for applying the knowledge in job conduct. The mental model is based on the blackboard metaphor (see Nii, 1986) and is ma& up of eight panels organi~cd into three categories. This organization supports efficient acquisition of new information and execution of sector control.

SECTOR WNAGEYENT

Sector Traffic I

Alrcraft \

Sector Control Evente Deta Plan

Aircmn Entering Sector

Potential Conflictiona

On-Going Ewne

Requests

Araa and Sector Weather Controll*r Factor.

Situmtion in Sector Thunderstorms

Turbulence

Temperature

Fectora

/ PREREQUISITE INFORMATION

SaCiOr AII~D.C* 1

Proc*dutea

Enmute Structura

Published Arrivals. Dwanures. Approaches

S W W U- A i i e

Topo~r.ph~

9.sar lhpe or Ha Spoa I FIGURE 4 Expert mental model of en route A'rC.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 15: Cognitive Task Analysis of Expertise in Air Traffic Control

At the top Icvcl, the menVal modcl contains scctor management, conditions, and prcrequisile information categories. The sector management category holds the dynamic knowledge of the %tor situation, the type of knowledge that is acccsscd and manipulalcd in working memory. The conditions category contains factors lhat influcncc general workload and the scleclion of spccific strategies for handling cvcnls. Thesc two categories contain information relaling to the situation in a specific xctor at a particular time, whereas the prerequisite information category contains knowledge of the sector and ATC procedures, strategies, and techniques that have bccn learned and committed to long-krm memory.

The organization of the scctor management category implies a spccific decision-making information flow. The controller lakes dala from the radar screen, flight progress slrips, and communication with pilots about individual aircraft. He or she processes these data about individual aircraft and categorizes them into events (a higher level construct involving one or more aircraft). By working wilh events rather than individual aircraft, the experienced controller can significantly reduce Ihe amount of information that needs to be manipulated in working memory, and can account for more aircraft to maintain better situation awareness. The hypothesis that expert controllers group dala by evenls is supported by the findings of the strategy analysis (discussed in the Analysis 3: Strategy Analysis section), in which experts tended to include more aircraft within each strategy. It is also consistent with the finding from the paired paper problem solving analysis that experts are more global in their analysis of problem solutions.

The long-term plan for controlling the sector is designed to handle events (represented as the primary and backup long-term plan levels within the sector control plan panel of Figure 4) that are used to store detailed plans for specific aircraft. This implies that the planning and kcision making involves events rather than individual aircraft. By working with procedures and strategies for cvcnt types, the amount of informalion needed lo make decisions can be significantly reduced.

The sector traffic event panel is the primary panel used for decision making about the priorilization of tasks and duties because it represents the understand- ing of the evenls that must be dealt with. Determining how lo deal with each event, however, involves reference to the data on the aircraft dab panel, the three conditions panels, and other events on the sector traffic events panel, as well as knowledge of standard and sector-specific procedures and strategies from thc procedures panel. The evenls are also interpreted with reference Lo the static spatial representation of the scctor airspace (embodied on the sector airspace panel).

Thc mental model inleracls with the lasks by providing information that triggers a task, and in return the task subgoal messages can provide new information to update specific mental-modcl panels. That relation is furlher quantified in Table 2, which lists how frequently specific panels interact with tasks either through triggers or subgoal messages being sen1 back to the mental madcl. This analysis shows which panels are likely to inleract most frequently with the controller tasks and

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 16: Cognitive Task Analysis of Expertise in Air Traffic Control

TABLE 2 Frequency of Interaction of Mental Model Panels With Controller Tasks

Mental Model Task

Categoty Panel Triggers Subgoals

Sector management Sector traffic events 21 4 Aircraft data 16 4 Sector control panel 3 4

Conditions Area and sector factors/weather factors 9 5 Controller factors 0 2

Prerequisite information Sector airspace 5 1 Procedures 1 1

subgoals. It is evident that changes in the sector traffic events and aircraft data panels are more likely to trigger a new task. Controllers are twice as likely to refer to these two panels as they are lo refer to the rest of the panels combined. In addition, the sector and weather factor panels are also likely lo receive updated information from subgoals. At present, no task-triggering information has been identified from thc controller factors panel, and this points to an area requiring further research.

Last, there are situational changes affecting the mental model that are external to the controller. These perceptual events are reflected through changes in the Plan View Display, flight strips, and communications from pilots or other controllers. Perceptual events consist of a trigger and a message that adds to the mental model. These perceptual events provide the mechanism to allow situational changes-such as an automated handoff or conflict alert-to directly affect task execution.

Validation. Validation was performed through the analysis of a timeline of task events that confirmed the mental model and the task decomposition. The most compelling evidence for the validity of the mental-model categories was that participants described their categorizations and depictions of sector events with the same level of specificity as the levels within the mental model. For example, the controllers used information about single aircraft and groups of aircraft to define events (e.g., conflictions, departures, arrivals, overflights, and requests). The salient aircraft data employed in Ule scenario were altitude, location, traffic typeiroute, and speed. This timeline analysis confirmed the structure and contents of the sector traffic events panel in several ways-for example, entrance, transit through, and exit from the sector were accurately depicted by the progression of levels in the sector traffic events panel from "aircraft entering sector" to "events nearing completion."

Validation was also obtained for the task decomposition, as the 13 tasks accommodated and accounted for all scenario events and controller operations. The performance of a task was initiated when the triggers for that task were present, which supports the validity of the task triggers.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 17: Cognitive Task Analysis of Expertise in Air Traffic Control

ANALYSIS 3: STRATEGY ANALYSIS

This section presents the analysis of controller cognitive-optimizing strategies as components of the controller's mental model. These strategies are heuristics that help the controller execute procedures more efficiently. Whereas controllerproce- dures are those collections of actions specified in the ATC handbook (FAA, 1989), cognitive strategies are less well-defined, can include a combination of procedures, are more difficult to verbalize, and may indicate controller expertise.

The cognitive-strategy analysis was made up of two parts. The first part, referred to as structured problem solving, was a comparison among expert, intermediate, and novice coatrollers working with problems at 65% complexity. These problems were designed so that the novices would not be overloaded; however, it was evident that the experts were not fully challenged. A new problem was designed for the experts, and this second analysis, called the work overload problem analysis, provides a more detailed look at expert-controller workload-management strategies.

In contrast lo the performance modeling procedure reponed in Analysis 2, which used only controllers' reIrospective reports, the controller-strategy analysis also used their concurrent reports. Recent restarch has found that concurrent verbaliza- tions accurately reflect the participant's thought, even under conditions of relatively automatic task performance (see Rhenius & Deffner, 1990).

Method

Participants. Five individuals from each group (expert, intermediate, and novice, as defined in the paired paper problem solving task in Analysis 1) partici- pated in the structured problem solving task. The work overload data collection was limited to 8 expert controllers with 4 or more years of FPLexperience, a mean age of 40.2 years (SD = 11.8 years), and a mean of 8.17 years of FPL (SD = 6.72 years).

Procedures. The participants were individually presented with structured problems at a simulated control console. Both structured problems were 20 min in length and were ratedat 65% complexity, with the first representing a job bottleneck and the second containing a number of time constraints. The problems were selected to reveal differing expert strategies under diverse control scenarios. The participants were f i t familiarized with the airspace and were then asked to solve the problems. They were recorded by video cameras and were prompted by the experimenter to describe what they were thinking about.

The work overload problem was designed to present a 125% workload for an individual working without assistance. This problem was specifically designed to challenge the expert controller. By Minute 14:OO in the scenario, there were about 19 planes in the sector with 5 arrivals, repxenting a number of ties and requiring substantial sequencing. The work overload problem required a degree of commu- nication that prevented Ule participant from verbalizing his strategies. Therefore

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 18: Cognitive Task Analysis of Expertise in Air Traffic Control

272 SEAMSTHI ET AL.

the problem was replayed, and the participant was asked to talk continuously about the situation, to discuss the control actions he made, and-most important-why he decided on that action. The participant was asked to be as explicit and detailed as possible in explaining the rationale for his decision to choose specific actions or strategies. During the playback, there were six freeze points at approximately 5-min intervals, and the participant was asked to describe what happened over the next minute in the sector by giving important information on the aircraft, the part of the plan being executed, and the specific tasks and strategies employed.

Analysis. For the structured problem solving, controller comments during the problem were transcribed using a selective notation procedure in which controllers' commenls about key a i d wen summarized (Redding et al., 1990). Each statement was then coded using the coding scheme listed in 'hble 3.

The coding scheme was developed as an extension of the strategies identified during Phase I. A list of the cognitive strategies was presented to 5 SMEs for rating and in-depth discussion. The SMEs were encouraged to discuss each strategy in relation lo the experience developed at their control facility. They were also prompted for related and additional strategies. The resulting strategy validation protocols were transcribed and reviewcd in order to ickntify additional strategies. Based on that review, the 22 cognitive strategies from Phase I were expanded to the 40 strategy categories listed in Table 3. These strategies were gmuped into the following three superordinate categories: planning, monitoring, and workload management.

Each selective notation was coded either as belonging to one or more of the listed strategy categories or as a procedural or non-strategy-related comment. The selective notations for all three groups were codedand the frequency of the different strategies was analyzed for both problems.

For the work overload analysis, the coding data were analyzed individually for each of the 5 expert controllers. First, the frequency of each strategy grouping was analyzed. Second, an SME was asked to note the errors made by each participant. Based on the error listings and each participant's comments about his own emrs, error frequencies were calculated for six problem time segments. The cumulative and detailed error frequencies were analyzed to determine their correlation with strategy usage.

Structured pmblems. A number of trends emerge in the frequency of strat- egy usage for both problems (see Figure 5). The experts tended to use fewer strategies than did intermediate and novice controllers. A review of expert and novice protocols revealed that experts tended to aocount for more aircraft in their strategies than did novices. Even though the experts showed lower strategy fre- quency, they used a greater variety of strategies. For example, the experts used 31 different strategies across both structured problems, whereas the novices used 26. Although the use of workload strategies was relatively infrequent, experts tended

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 19: Cognitive Task Analysis of Expertise in Air Traffic Control

TABLE 3 Strategy Codes Used in the Analysis

Planning Strategies Are there conflictions or potential conflictions? Detmnine aircraft requirements Determine amount of time available to effect separation once h a f t is in sector Determine f o m of sepamtion (e.g., vertial or lateral separation) Determine how weather and winds will affect the sector

/ Determine sequence Determine the nature of the overtake Determine when to implement backup plan Determine when to start an action Determine which airaaft to make first Develop backup plan Develop early primary sector plan Does the aircraft require special attention? Letting speed take effect Prioritize actions Refiie and update primary sector plan or action plan Wait and see What are the airaaft variables, including altitude, speed, route, and traffic? What are the aircraft's performance class or characteristics? Which action can be completed the quickest?

Monitoring strategies Evaluate adjacent secton Monitor to start action Monitor action to completion Monitor separation Monitor sequencing Monitor to compare strips with Plan Vim Display data Monitor to review and update control action plan Monitor to update primary sector plan or implement backup plan Monitor to vector airaaft Monitor to verify aircraft has reached altitude

Workload managment strategies Are there times of heavy sector traffic and workload? Will descending an airaaft achieve the quickest separation? Determine action requiring minimum coordination Determine how to expedite aircraft through your sector Determine which action results in the lower workload Determine what to do to eliminate a factor Identify aircraft that are not a factor Is it efficient to assume early control (i.e., reaching out)? Monitor workload Select action that will require least monitoring

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 20: Cognitive Task Analysis of Expertise in Air Traffic Control

140

1 20

100

Number of eo Stratogler

60

40

20

0

FIGURE 5 Frequency of strategy usage for both strudured problems

to use them more frequently than did intermediates and novices. It appears that the experts may be more efficient at managing a sector with fewer

strategies. This finding supports the idea inherent in the COGNET mental model that experts organize aircraft into events. Therefore, it is possible that many of the expert strategies are implemented at the event level rather than at the individual aircraft level.

A 2 x 2 (Experience Level x Problem Type) analysis of variance (ANOVA) showed a significant effect of experience level on the use of workload manage- ment strategies, F(2, 24) = 20.39, p < .0001. Post hoc comparisons showed significant differences between all three experience levels, with experts having the greatest frequency of workload management strategies, followed by inter- mediates.

A 3 x 2 x 3 (Experience Level x Problem Type x Strategy) ANOVA was performed to analyze the number of strategies across both problems. There were significant main effects for problem type, F(l, 72) = 21.52, p < .0001, and strategy, F(2,72) = 1243.92, p c .0001. Several interesting patterns emerged between strategy usage across the two structured problems. When working on the job bottleneck problem, experts used significantly fewer @ c .05) planning strategies than did intermediates and novices. On the time-constrained problem, experts and intermediates used significantly more @ c .05) workload manage- ment strategies.

In general, experts used significantly more workload management strategies but relatively few planning strategies. Their less frequent use of planning strategies may reflect the fact that their experience made it unnecessary to do as much planning. An examination of specific strategy usage revealed important differ- ences. The strategies "Identify aircraft that are not a factor" and "Determine how to expedite aircraft through your sector" were used more often by expert control- lers. However, the planning strategy "Determine when to start an action" was used

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 21: Cognitive Task Analysis of Expertise in Air Traffic Control

ATC COGNITIVE TASK ANALYSIS 275

more frequently by novices. Thus, experts did make more use of workload man- agement strategies. These strategies serve to simplify the situation and decrease workload, thus reducing the monitoring effort required.

Work overload problem. Eight expert controllers solved the work overload problem. As shown in Figure 6, these experts used an average of 123.6 strategies in solving the 20-min work overload problem. Compared lo the previous structured problem solving, experts in this analysis used workload management strategies more frequently (22%).

Correlations were calculated between the number of controller errors and the number of planning, monitoring, and workload management strategies used by the 5 experts (see Table 4). The correlation between number of errors and monitoring strategies (r = -.912, n = S), is significant @ < .05). This correlation demonstrates that, when more monitoring strategies were verbalized, fewer errors were made during the problem. Although the correlations between errors and planning strategies (r = -.69, n = 5) and between errors and workload management strategies (r = -.63, n = 5) were not significant, the same trend toward a negative correlation is evident. Therefore, the results show that experts who made fewer errors were more likely to use more strategies in the manage- ment of their air traffic sector.

Discussion

Experts tended to use fewer strategies than did less experienced controllers. One explanation for this finding is that experts tended to account for more aircraft in

Individual Participants

FIGURE 6 Strategy and comment frequencies for work overload problem.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 22: Cognitive Task Analysis of Expertise in Air Traffic Control

h)

2 TABLE 4 Relationships Among the Constructs

Task Task Perceptual MM Triggers Subgoals Events Strategies

MM - MM contents - Facilitates Contents of MM provide awareness of conditions information peceptual events category is needed for switch~ng triggers mechanism for

strategy use - - Trigger execution of

task subgoals Task Cause controller to

triggers execute tasks, resulting in update of MM

Task Performance of ., May trigger task by subgoals @the subgoals adding message to

adds messages to MM MM level

Perceptual Allow for May trigger task by events situational adding message to

changes to update MM MM directly.

May affect strategy use by adding message to MM Conditions category

-

independent of tasks

Strategies Reside in Procedures Used to execute task panel, primarily subgoal General Strategies level

Note. MM = mental model.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 23: Cognitive Task Analysis of Expertise in Air Traffic Control

each strategy implementation. Experts also displayed a wider repertoire of strate- gies as indicated by their verbal reports. Compared to novices, experts showed a greater frequency of workload management strategies, which can simplify the management of the sector by reducing the number of aircraft that the controller must attend.

The results also show the utility of using problems with varying characteristics to elicit the full range of expert smtegies (see Hoffman, 1987).

COMPARATIVE RESULTS

The results from the three analyses have been combined into an integrated frame- work of en route ATC expertise. Table 4 lists the principal relations among the results of the CTA. The mental model provides the framework for maintaining situation awareness, in which tasks are performed with reference to the current state of the mental model. As a hmework, the mental model interacts with domain knowledge, task subgoals, task triggers, perceptual events, and strategies. The task triggers activate tasks, which results in an update of the mental model thmugh task completion or subgoal messages. Perceptual events and communications fkom pilots, other controllers, or the radar also provide updates to the mental model. Therefore, the mental model is the central agent in sending and receiving messages from the supporting enti ties.

Although the mental model is based on a blackboard paradigm, it does map to an information processing model of the human operator. The Wee high-level categories of the mental model correspond to psychological constructs. The sector management category of panels relates to situation awareness in working memoq. Maintaining situation awareness is done primarily thmugh working memory with access to long-term memory for stored information (see Sarter & Woods, 1991). The conditions category operates primarily as a switching mechanism when the controller is experiencing high workload that calls for a different set of procedures and strategies. The prerequisite information category in the mental model includes the procedures, structures, and knowledge in the controller's long-term memory that are required to control a sector.

The controller tasks are linked to the mental model through the task triggers and task subgoal messages for updating the mental model. The primary cognitive tasks-maintain situation awareness and develop and revise sector control plan- are tightly coupled with the mental model. The mental model is frequently updated through the performance of these two tasks, resulting in changes in the model messages. These changes may trigger the performance of a task if the message matches one of the task's triggers. When a message matches a task trigger, the controller performs the appropriate task if its priority is the highest of all triggered tasks.

The controller strategies are linked both to the mental model and to the controller tasks. The strategies may be thought of as part of the mental model and are included in the procedures panel as general and sector-specific strategies. Although knowl-

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 24: Cognitive Task Analysis of Expertise in Air Traffic Control

edge of the stralegies resides in lhe mental model, the strategies are activated by task subgoals. The task subgoals combine with the workload conditions in the menlal model and with critical cues to activate the appropriate strategy for the given conditions. Although all tasks have linkages to strategies, the primary linkage is between the strategies and the maintain situation awareness and develop and revise sector control plan tasks.

The paired paper problem solving adds support lo the mental model in several critical respects. Results from this analysis indicate that experts look a more comprehensive and organized view of the evolving situation. This suggests the presence of an underlying mental model. Additional paired paper pmblem solving results show that experts were more adept at initially perceiving all the important events in a pmblem. This would aid the expert in decomposing the sector into sector events. Finally, the results from Analysis 1 indicate that experts used workload reduction methods (e.g., compuler entry) most often. This supports the strategy analysis, which concluded that experts are characterized by an increased use of workload management strategies.

There is support for a key aspect of this framework from research done on the en route controller's communication (Seamster et al., 1991). This research included a &tailed analysis of controller-learn communication and communication between the radar conlroller and the associate radar controller. The results show thaf when controllers worked as a team, the radar controller used a substantial number of situational inquiries to help maintain situation awareness. The associate radar controller complemented that activity by providing situation awareness informa- tion to the controller through the use of observations and answers that supplied information. The radar controller also communicated the sector control plan by using a greater number of statements of intent that responded to the associate radar controller's inquiries about intent. These results combined to demonstrate that more than 80% of controller-4eam communication concerned the two primary cognitive tasks: mainlain situation awareness and develop and revise sector control plan. These results support the emphasis p l a d on these two cognitive tasks by the current CTA, and substantial effort should be expended toward training controllers in these two tasks. Their imporlance may suggest implications for student selection as well.

DISCUSSION

Expert air traffic controllers possess a complex knowledge base of ATC con- cepts, principles, procedures, regulations, and strategies. The controller's men- tal model contains a number of characteristics that allow the controller to circumvent the limitations of normal human information processing (Salthouse, 1991), a chief barrier to expertise in time-constrained, multitasking environ- ments. In this view, expertise promotes a type of performance that is not constrained by the type of information-processing limitations encountered by novices and nonexperts.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 25: Cognitive Task Analysis of Expertise in Air Traffic Control

The expert controller's dynamic mental model provides not only an eficient knowledge structure, but also a means for the rapid retrieval of sedor information. This rapid retrieval is demonstrated by the faster nsponse times found in olhcr areas of expert performaace (e.g., typing and Morse code operation), and is one of the key attributes of superior or expert perfkxmamx (Ericssoa & Smith, 1991). In addition to rapid retrieval, tbc mental model provides a mechanism for categosizing aircraft into important sexlor ewents. This grouping allows the expert (a) to work with more aircrafl, @) to better formulate a sector plan, and (c) to use fewer control actions and strategies in the control of the sector. Expert p p i n g of airaaft is supported by research in the recall of seaor aircraft (!%hger, Means, & Roth, 1990), which demcmtmled that expericncedconwllcrs r e c a l l c d ~ ~ c airaaft in p u p s of two to four aircraft. It is interesting to note that the aircraft pupings were based on potential conflicts and/or required control actions rather than on aircraft proximity.

The mental model also has a built-in prioritization scheme that gives priority to aircraft altitude, location, and mute. This provides the expert controller with the knowledge of key aircraft data elements. Additionally, the expert's mental model contains a set of panels that evaluates conditions relating to the overall sector, weather, MIC volume, and personal factors. These panels help the expert to evaluate key conditions and to switch to workload reduction strategies under heavy workload. Thus, the mental model has mechanisms for prioritizing controller actions and for managing workload

These four characteristics of the expertcontroller mental model constitute some of the primary components of A X expertise and underscore some of the key elements of expertise in time-constrained, multitasking environments. Expertise in these environments requires the knowledge structures as well as the elements within the mental model that promote fast performance. In the case of air traffic controllers, there are at least four mechanisms that contribute to fast performance: (a) The mental model provides a structure for efficient and rapid retrieval; @) the mental model encourages the categorization of aircraft into sector events, allowing more efficient and rapid planning of controller actions; (c) the model provides for the prioritization of aircraft data elements, allowing the expert controller to operate more efficiently; and (d) the mental model has a sophisticated mechanism for workload management-+ critical aspect of task execution in timeconstrained environments. The mental model includes a range of workload management strategies that encounge the expert to determine when an aircraft is not a factor or to determine how to expedite an aimaft through the sector. The mental model also helps the controller evaluate the sector conditions to determine when to switch to more efficient strategies in the presence of work overload These four mechanisms appear to be significant attributes of expertise in complex, timeconstrained, multitasking environments.

Training Implications

ATC is a complex set of skills and howledge, and students must be provided with some way to overcome the normal information processing limitations. Based on

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 26: Cognitive Task Analysis of Expertise in Air Traffic Control

the results of this CIA, me student should be provided with a framework for acquiring, organizing, and utilizing KTC knowledge. The mental model of the expert controller can provide a structure for learning as well as for the efficient performance of en route KTC. Teaching this structure to students will provide them with access to expert knowledge by helping them to organize information into a more accessible knowledge base. The mental model also provides a framework for acquiring and interpreting data about the sector situation and for making decisions about how to control the sector. The structure of the mental model, with its emphasis on sector events, provides a way to group aircraft and strategies so that they can be learned more efficiently and accessed more readily during ATC.

The controller's mental model can provide the framework for more efficient training, and should be the basis from which skills and knowledge are taught. We recommend a series of model-building exercises in which the student is allowed to refine his or her model until it approximates the experts' mental model. Based on the mental model, controller procedures andevents should be taught by event type. To do this, training should emphasize the integration of sector aircraft information into sector-relevant groupings so that students can see important relations between aircraft. This event-based appach includes training in the recognition of event types and in the categorization of aircraft into appropriate events. In addition, this approach should train controllers to scan for sector events. This type of scanning can improve memory efficiency by organizing sector data into chunks of related information.

The mental model also provides the framework for training in the recognition of task triggers. Rapid recognition of task triggers is critical because they specify the tasks and operations that the controller must accomplish at a particular point in time. Also, implicit in the triggers is the prioritization scheme for task performance. Training in task triggers should be provided through part-task training, in which students are given repeated practice in recognizing and identifying task triggers. This training should include practice in trigger recognition as well as in the understanding of the relation between the mental model, controller tasks, and task triggers.

The results indicate that the two cognitive tasks (maintain situation awareness and develop and revise sector control plan) should receive primary emphasis in training. These two tasks should be taught and practiced in an integrated manner so that they support the performance of the other 11 controller tasks. By emphasiz- ing the cognitive subgoals within each task, training can concentrate on the effective integration of the cognitive operations into the procedural sequences for controller task performance. Situation awareness is a particularly important skill in effective AXC, and a computer-based training program has been designed to teach global situation awareness skills to students during the intermediate and latter phases of controller training (see Redding, 1992a). Students must learn to maintain situation awareness by updating their mental model after execution of each task subgoal, and this is one of the objectives in the situation awareness training block. Students are presented with a series of scenarios of gradually increasing complex- ity, each involving a particular controller task (e.g., manage arrivals). Using a

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 27: Cognitive Task Analysis of Expertise in Air Traffic Control

modified situation awareness global-assessment technique (Endsley, 1988), when- ever the student completes a key task subgoal the scenario is frozen and all displays are blankedout. The student must t k n describe the current state of his or her mental model, and feedback is then given through comparisons with the actual scenario and with expert model descriptions.

The training program based on the results from these cognitive analyses orga- nizes instructional objectives around the key characteristics of the expert mental model and strategies (Redding, Cannon, & Seamster, 1992). This does not mean that students are taught to structure their knowledge and skills around a fixedexpert model. In fact, certain aspects of novice models may be helpful for students during early stages of learning by providing a useful preliminary organizer. Novices, for example, might classify aircraft data into different traffic event types than would an expert. During the early stages of instruction, it is important that students practice thinking about aircrafl in terms of events; they will learn the relevance of the different event types later.

The resulting training program is organized into blocks, with the instruction in each block tailored so that small chunks of related knowledge and skills are taught together. The instruction is immediately followed by simulation-based practice. Using this problem-based approach to instruction, simulator exercises provide the context for assimilating basic &main knowledge. Implicit in the simulator exer- cises are the primary organizing principles of controller expertise. Studenls are presented with a series of limited control scenarios that let them see the underlying commonalities and predictable elements across various situations involving a particular type of sector event, task, or problem. Instruction begins with mental model development so that training can promote both the development of efficient organization and the acquisition of domain knowledge. In each stage of learning, the mental model guides and organizes learning activities and is elaborated throughout training. Planning skills are then taught in relation to the mental model, followed by procedural knowledge. Once students have developed an effective mental model and have mastered basic planning and ATC tasks and skills, they practice situation awareness skills in relation to the mental model and controller tasks. Finally, students refine and extend their skills through training based on critical and abnormal incidents.

ACKNOWLEDGMENTS

This report was prepared as an account of work sponsored by an agency of the U.S. Government. This research was sponsored by the Federal Aviation Administration under Contract OPM-87-9041 with the U.S. Office of Personnel Management, Training Management Assistance Division. Richard E. Redding was the director of the project and the principal investigator. The views and opinions of the authors expressed in this article do not necessarily state or reflect those of the U.S. Government or any agency thereof.

We thank K. Anders Ericsson and Thomas B. Malone for their helpful comments

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 28: Cognitive Task Analysis of Expertise in Air Traffic Control

282 SEAMSTER ET AL.

on a draft of Ulis work. Special Ulanks are due to Bruce C. Lierman for data collection and analysis of the paired paper problem solving. Finally, we thank the FAA En Route Cu~~iculum Redesign Project team members for their support and assistance.

REFERENCES

Adelson, B. (1981). Problem solving and th development of alstncl categories in programming languages. Memory & Cog* 9,422433.

Cannon-Bowers, J. A, Tannenbnum, S. I., Was, E, & Converse, S. A. (1991). Toward the integration of tnining theory and technique. H w ~ n Factors, 33,281-292

Card, S. K., M o m , T. P., & Newell, A. (1S3). The prychobgy of human-ccmrpuer inferaction. Hillsdale, NJ: Lawrena Mbaum Associates, Inc.

Chi, M. T. H., Feltovich, P. J., & GI-, R. (1981). Categorization and representation of physics problems by exputs and novices. Cognitive Science, 5, 121-152

CTA lncopnted. (1990). NLA air tr+ c o n h ~ l ojxratiars umcepts: VoL W. ARTCC-hast ar route controllers (DOT/FAA/AF'-87-01). Was4iyt00, DC: Federal Aviation Administration.

de Groot, A. D. (1965). Thou& a d dmke in k. The Hague, The Netherlands Mouton. Endsley, M. R. (1988, May). S h i m awreness global arsrrnnu~ technique (SAGAT). Paper

at Ihe National kvoqmce and EJectronic Conference, Dayton, OH. Ericsson. K. A., & Smith, 3. (1991). Prospects and limits of th empil id study of expertise: An

in tduc t ion In K A Ericsm & J. Smith (Eda), Toward a gmeml theory ofcrpcnke: Pmpectr and l i d s @p. 1-38). New Ywk: Cambridge University Press

Feded Aviation Administration (1989). Air t+ c m d (Order No. 7110.65F). Washington, DC: Author.

H o h a n , R. R. (1987). The problem of exlracting the lolowledge of expetts from the perspective of experimental psychology. AX M a & q 6(3), 5-7.

Means, B., & Go& S. P. (1988). Cognitive teclr analysis as a basis for tutor development: Articulating abstract knowledge qmsentatiolrp. In 3. Psatly D. Masey, & S. A. Mutter (Eds.), Inl~Iligen~ hcloring syslcn~: Le- l a r d @p. 35-57). Hillsdale, NJ: hwmnce Erlbaum Associates, Inc

Nii, P. H. (1986). Blacklumd systems: Pt. I. Ibe blackboard model of problem solving and the evolution of blackboard arcbitedure. AI Magazine, 7(2), 3S53.

R&g, R. E. (1992a). Analysisof opera&bnal urorsandworkloadin air trafficcontrol. In Proceeding ofthc36h ~~~olmce&goftheH~~nFaclor~SociCty @p. 1265-1269). Santa Monica, CA: Human Factors Society.

Redding, R. E (1992b). A stan&rd poordvre for conducting cognitive task a ~ l y s k : Reporl to the FedemlAv ia t ion~k t ro t im (ERIC Document Reproduction Service No. ED 340847). McLean, VA. Human Technology, Inc

Redding R. E, Cannon, J. R., Lierman, B. C, Rydu, J. M, Swmsler, T. L., & Purcell, J. A. (1990). Cognitiw rosk analysis qfprioritiz&on in air t r a m cor~ml: Report to the FBdoml Aviation A&niakrratiOn McLean, VA: Human Techology, h. (ERIC Document Reproduction Service No. ED 340 849)

Redding, R. E, Cannon, J. R., & Swnster, T. L (1992). Expertise in air traffic conlrol (ATC): What is it, nnd bow can we train for it? In Pmcesdings ofthe 36th a ~ w l nrating of h e Human Factors Soeiety @p 1326-1330). Spnta Monica, CA: Human Factors Society.

Redding, R. E, Ryda, J. M., Seamster, T. L, Purcdl, 3. A., & Cannon, 3. R (1991). Cogdive raPC analyskofen m r & a i r t m f i c ~ : Moddextemion andw&ion:Report tothcFedemlAviation Albninistratiar McLean, VA. Human Technology, Inc. (ERIC Document Reprodudion Service No. ED 340 848).

Redding, R. E, & Seamster, T. L. (in press). Cognitive task aaalysis in air lraffic controller and aviation

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14

Page 29: Cognitive Task Analysis of Expertise in Air Traffic Control

crew mining. In N. Johnston, N. McDonald, & R Fuller (Eds.)., Aviclrion prychology in practice London: I#gate.

Reitman, 3. S. (1976). Skilled puception in Go: Deducing memory seuctlues born inter-response times Cognitive Psychology, 8,336356.

Rbmiug D, & D e 5 , G. (1990). Evaluation of concumnt thinking aloud wing ye-eodring data. In Pruceedings of the 34th sand mding of & H- Fauors Sociciy @p. 1265-1269). Santa Monica, CA: H u m a n F ~ Sodety.

Rydq J. M., & Redding, R E (1993). Integntine cognitive task analysis into ipptruUio~I systems amlopment w h d Tockndogy & k b p a t t , 41(2), 75-96.

Salthouse, T. A. (1991). F k p d ~ ~ as the circumvention of human pwxsbg limitations. In K. A. E~-S & I. smi* owo or do -1 thewy ofrxpnrirr: ~mopccu ~d w @p. 286-300). New York: Cambridge U M l y Pms.

Sarter, N. B.. & Woo&, D. D. (1991). Situation awuuuss: Acritiul but ill-defioed pbe~iomenon. The I n t e d l Jounml ofAvicuion Psychdogy, 1,4547.

Samster,T.L,~m,J.R,Ruall,J.k,Piera,R.M.,Fisher,S.G.,&~~R.E.(1991). Anotysis of cuntrdlcr . . ine~~mruairtr&conlrd:Re~~rt&theFehrcllAvirrrion A d n i t b e Whan, VA: Human Technology, Inc. (ERIC DoaMent Repdudion Service No. ED 341 091)

Scbhger, M. S., Meaq B., & Roth, C (1990). Cbpitive task analysis for the real(-time) world. In Pruc&gs of the 34th d nwr8in.g of the Human Factors Society @p. 1309-1313). Santa Monica,CA:Hump.FoawSociety.

Wilson, 1. R, & Rulberfonl, A. (1989). Mental models: Tbeory and application in human factors Hvmon Factom, 31,617434.

Zachary, W., Ryder. I. hi., ROSS, L, & Milad, M. Z (1992). Intelligent compter-bumpn interaction in real-time, multi-tasking poass axtrol and monitoring systuns. In M. Helruder & M. Nagamichi (EdP.), H w ~ n factors in &rip man~#acmtubility @p 377-401). Taylor & Frauag Lid.

Dow

nloa

ded

by [

The

Uni

vers

ity O

f M

elbo

urne

Lib

rari

es]

at 0

4:30

13

Sept

embe

r 20

14