effects of modes of cockpit automation on pilot performance and

Effects of Modes of Cockpit Automation on PilotPerformance and Workload in a Next GenerationFlight Concept of OperationGuk-Ho Gil,1 David Kaber,1 Karl Kaufmann,1 and Sang-Hwan Kim2

1 Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, North Carolina2 Department of Industrial and Manufacturing Systems Engineering, University of Michigan-Dearborn, Dearborn, Michigan

Abstract

The objective of this study was to compare the effects of various forms of advanced cockpit automationfor flight planning on pilot performance and workload under a futuristic concept of operation. Alab experiment was conducted in which airline pilots flew simulated tailored arrivals to an airportusing three modes of automation (MOAs), including a control-display unit (CDU) to the aircraft flightmanagement system, an enhanced CDU (CDU+), and a continuous descent approach (CDA) tool. Thearrival scenario required replanning to avoid convective activity and was constrained by a minimumfuel requirement at the initial approach fix. The CDU and CDU+ modes allowed for point-by-pointpath planning or selection among multiple standard arrivals, respectively. The CDA mode completelyautomated the route replanning for pilots. It was expected that the higher-level automation wouldsignificantly reduce pilot workload and improve overall flight performance. In general, results indicatedthat the MOAs influenced pilot performance and workload responses according to hypotheses. Thisstudy provides new knowledge about the relationship of cockpit automation and interface featureswith pilot performance and workload in a novel next generation–style flight concept of operation.

C© 2012 Wiley Periodicals, Inc.

Keywords: Level of automation; Modes of automation; Cockpit automation; Flight simulation;NextGen, Cognitive workload

1. INTRODUCTIONIn a recent report on fatalities in aviation accidents, lossof control in flight was found to explain 65% of all ac-cidents (Boeing, 2008). In a number of historical cases,loss of control has been attributed in part to pilot “out-of-the-loop” performance problems and loss of situa-tion awareness due to high-level cockpit automation.

Correspondence to: David Kaber, Edward P. Fitts Departmentof Industrial & Systems Engineering, North Carolina StateUniversity, Raleigh, NC 27695-7906. Phone: 919-515-0312;e-mail: [email protected]

Received: 5 January 2011; revised 27 July 2011; accepted27 July 2011

View this article online at wileyonlinelibrary.com/journal/hfm

DOI: 10.1002/hfm.20377

For example, regarding the American Airlines Flight965 incident in Cali, Colombia, Endsley and Strauch(1997) reported that features of the automated aircraftflight management system (FMS) contributed to piloterror. Specifically, pilots were not aware that deletion ofwaypoints from a preprogrammed flight path using amulticontrol display unit interface caused vertical pathcontrols to be disabled. This ultimately contributed tothe crash.

In general, when a pilot must replan a route be-cause of either non-nominal (e.g., runway closure orchange, weather disturbance) or emergency (e.g., cargofire, medical emergency, etc.) circumstances (Kalambiet al., 2007), there are required functions, includingsystems monitoring, generating decision alternatives,and selecting and implementing options. The functions

Human Factors and Ergonomics in Manufacturing & Service Industries 00 (0) 1–12 (2012) c© 2012 Wiley Periodicals, Inc. 1

Effects of Modes of Cockpit Automation Gil et al.

actually performed by the pilot depend on the level ofaircraft automation. Levels of automation in human-in-the-loop systems define function allocation schemesfor the human and machine (Endsley & Kaber, 1999).Pilots using low-level cockpit automation may need toperform route selection or implementation functionsas compared to using high-level automation, wherethey may need to perform only route generation andmonitoring functions. In general, lower level automa-tion may increase pilot workload, but high-level au-tomation may result in a loss of pilot awareness of thestates of the aircraft or airspace, particularly duringlow workload phases of flight. A loss of pilot situationawareness in replanning can lead to errors and perfor-mance problems.

In a reroute scenario, the flight path of an aircraftis typically extended, and pilots must effectively mon-itor aircraft position changes to prevent flight into ter-rain, fuel consumption to ensure sufficient levels forflight before landing, and weather conditions to pre-vent damage to the vehicle. A loss of situation awarenessin a reroute situation due to the occurrence of non-nominal or emergency conditions can also increasepilot workload as a result of a need for pilots to quicklyreorient to the state of the automation and then tobegin to process information (Sarter & Woods, 1995).

Previous research (Chen & Pritchett, 2001; Kaberet al., 2009; Kalambi et al., 2007; Wright, Kaber, &Endsley, 2003) has investigated the effect of variousforms of cockpit automation on pilot performance andworkload in replanning scenarios. The use of high-levelautomation for flight path control (e.g., the FMS) hasbeen found to support superior piloting performancein advance of critical events, such as an approach re-vision (Chen & Pritchett, 2001; Wright et al., 2003).However, the use of low-level automation, engagingpilots to a greater extent in aircraft control loops, hasbeen found to support superior pilot situation aware-ness and performance in dealing with an event, like anapproach revision. Related to this, Chen and Pritch-ett (2001) investigated the effect of different levels ofautomation on generating and predicting a detailedtrajectory to the runway threshold in a simulated emer-gency situation. Under the highest level of automation,pilots exhibited automation bias (i.e., tending to followthe automatically generated trajectory because of highworkload and time pressure), even when the tool gen-erated erroneous trajectories (Chen & Pritchett, 2001).Similarly, Wright et al. (2003) observed that pilots us-ing high-level automation placed less emphasis on the

accuracy of flight planning in advance of critical eventsand greater reliance on data stored in the FMS andthe automation capability. In a related study, Kaberet al. (2009) reported that when pilots used high-levelcockpit automaton (the FMS) in advance of receiving aclearance for an alternate runway in a simulator-basedlanding scenario, they were more likely to forget criticalprocedures (e.g., backcourse for opposing instrumentlanding system runways) and to make errors in flightpath control, as compared with when they used low-level automation. However, Kaber et al. (2009) alsofound that the use of high-level automation led to sig-nificantly lower objective and subjective pilot workloadresponses. Still other studies have demonstrated thesame workload advantage of high-level (FMS-based)cockpit automation in simulated critical event (flightreplanning) scenarios (Kalambi et al., 2007). It is im-portant to note here that, in some investigations of theeffects of levels of cockpit automation on pilot perfor-mance, higher levels of automation have also providedpilots with less complex and more usable interfaces ascompared with modes of low-level automation, whichmay also contribute to reduced workload experiences.

Consequently, the design of contemporary advancedaircraft automation has been focused on the opportu-nity for reducing pilot workload but, at the same time,attempting to prevent the potential for loss of situationawareness in high-workload phases of flight, includ-ing airport approach and landing. Continuous descentapproach (CDA) tools provide pilots with the capabil-ity to define an airport arrival at an increased distancefrom touchdown without the typical requirement of al-titude “step-downs” and with the use of low continuouspower throughout the descent (Coppenbarger, Mead,& Sweet, 2007). Coppenbarger et al. (2007) investigatedthe Tailored Arrival (TA) concept for enabling contin-uous descents under constrained airspace conditionswith a datalink (DL)-enabled CDA. Continuous de-scent arrivals are “tailored” in the sense that the arrivalis constructed for the pilot to take into account airtraffic control (ATC) constraints, such as traffic pat-terns, airspace restrictions, and minimum altitudes, aswell as the performance of the specific aircraft type. Aclearance for a TA can be given well in advance of thepoint at which an aircraft begins its descent becauseroutes are predetermined and have been coordinatedacross ATC facilities. From the pilot’s perspective, theTA means far fewer flight tasks and actions at cockpitinterfaces. It also allows for more time to deal with anyoff-nominal events that might develop. Beyond this,

2 Human Factors and Ergonomics in Manufacturing & Service Industries DOI: 10.1002/hfm

Gil et al. Effects of Modes of Cockpit Automation

TA clearances are generally given by DL (electroniccommutation between a control center and the air-craft) to further reduce flight crew and ATC workload.(Although pilot performance effects of linkages of DLto the FMS of an aircraft are also important to under-stand in such scenarios, the focus of the present workwas on assessment of the impact of various forms ofcockpit automation for route replanning and not forconveying information from ATC or a CDA-like toolto the FMS.) In general, the TA scenario requires ahigh-level of aircraft and ATC automation for effectiveperformance.

Very recently, Dao et al. (2010) investigated the ef-fects of automated spacing tools during a CDA on pilotperformance, including altitude, speed, and time-in-trail. However, no previous studies have empiricallyassessed the impact of CDA tool use on both pilotworkload and performance (based on flight situationawareness) in comparison with conventional formsof cockpit automation, including the FMS. It is im-portant to note that other prior investigations in thedriving and military domains have used and evaluatedmeasures of operator performance as a basis for non-intrusiveness capture and analysis of situation aware-ness (Kaber et al., 2005; Ward, 2000). The objectiveof this study was to compare the effects of varioustypes and levels of cockpit automation for flight plan-ning, including a CDA tool, on pilot workload, time-to-task completion (TTC), and decision making in anext-generation (NextGen)-style TA that followed theCDA model. NextGen is the Federal Aviation Admin-istration (FAA) and National Aeronautics and SpaceAdministration (NASA) acronym for referring to tech-nologies and procedures to be developed as part of theNext Generation Air Transportation System with animplementation timeline extending approximately tothe year 2020.

2. METHODS2.1. Flight Simulator Setup

A high-end, PC-based flight simulator (see Figure 1),integrating X-Plane simulation software, was set upto present a B767-300ER cockpit with interfaces andfunctions of existing and futuristic forms of cockpitautomation, including a control-display unit (CDU)for the FMS, an enhanced CDU (or CDU+) withpredetermined route selection capability, and a CDAtool. Each type of automation was integrated with anATC DL system for electronic relay of clearances be-

Figure 1 Flight simulator setup.

tween the simulated cockpit and control center. In ad-dition to three MS Windows–based PCs, the simulatorsetup consisted of a cockpit workstation with flightdeck controls, including a yoke, a throttle quadrant,and rudder pedals (see Figure 1). Two LCD moni-tors were arranged vertically with the lower displaypresenting the instrument panel for the B767-300ER,including the PFD (primary flight display), FCP (flightcontrol panel or glare shield), and FMS with theCDU interface. The upper display presented an out-of-cockpit view. The X-Plane simulator software sup-ported presentation of dynamic flight situations withhigh-fidelity graphics. A touch screen monitor wasused to present the new CDA planning tool (see rightside of Figure 1). Pilots used the screen to directly ma-nipulate graphical buttons and select route waypointsor entire routes determined by the automation. Thedisplay content of the monitors was synchronized us-ing a TCP/IP (Transmission Control Protocol/InternetProtocol) network supported by X-Plane.

2.2. CDA Tool Development

The CDA tool is a contemporary form of cockpit au-tomation originally retrofitted in the UPS Boeing 757and ABX Boeing 767 aircraft. Unfortunately, the cur-rent version of the X-Plane simulation software doesnot provide a CDA interface. For this reason, a custom

Human Factors and Ergonomics in Manufacturing & Service Industries DOI: 10.1002/hfm 3


SCRATCH PAD

Figure 2 CDA tool interface design.

CDA tool was developed and integrated with the flightsimulator. It includes a common user interface (CUI,see Figure 2a), a navigation display (ND, see Figure 2b),and a data grid (DG, see Figure 2c). In the experiment,pilots interacted with the CUI for receiving/requestingATC clearances. The ND also provided weather infor-mation with color coding (e.g., red = severe weatherconditions; blue = mild weather conditions). The CUIand ND are located in the middle of the cockpit dis-play area. The DG is located below the CUI and ND.When a pilot uses the CDA to confirm a new routeto avoid severe weather conditions, address fuel con-straints, and so forth, the CDA sends route informationto the X-Plane software. X-Plane then loads the newroute into the CDU/FMS. In this way, the CDA toolprovides pilots with the capability for a TA.

2.3. Scenario Development

A realistic flight scenario was developed to support theobjective of empirically assessing the impact of cockpitautomation on pilot performance under high- and low-workload conditions. The Reno-Tahoe InternationalAirport (KRNO) was selected as the termination of anapproach with reroute followed by landing. The airportis located in a valley with terrain rising to approximately6,000 feet above the field, making arrivals challengingfor pilots even when using cockpit automation. Theextreme terrain makes maintaining the published flightpath more critical than when operating at most otherairports and increases the utility of automation forpilots.

The scenario required a pilot to use all features ofthe CDA tool, including the ND to perceive and plota route around a thunderstorm, the DG to determinefuel consumption for the reroute, and the CUI and DLto communicate with ATC for clearance of the newroute. After confirming the new flight path with ATC,the pilot needed to perform different types of actionsto change the flight path.

The scenario began with the simulated aircraft fly-ing westbound toward a radio ground station approxi-mately 150 nm (nautical miles) east of the airport [theWilson Creek VOR (Very High Frequency/VHR Omni-directional Radio-beacon)]. The pilot received a DLclearance for the TA from ATC, with the route appear-ing automatically on the ATC uplink display. The pilotreviewed the route focusing on whether it was flyableunder the constraints of weather and fuel consumptionlimits. (In all trials, the first clearance was to be rejectedbased on weather conflicts or fuel constraint violations.The pilot then had to replan the route and seek approvalfrom ATC.) When an acceptable clearance was received,the pilot confirmed this via the DL using a button onthe planning tool touchscreen. The TA route was thenloaded into the CDU/FMS either by data connectionbetween the DL system and FMS (for the CDA tool orCDU+ mode) or manually (for CDU mode).

2.4. Independent Variables

There were three independent variables, including themode of automation (MOA); the level of flight work-load, manipulated in terms of the time available to



perform the reroute planning; and the phase of flightinvolving planning and decision making or flight planimplementation. The MOAs were CDU, CDU+, andCDA. The automation under all modes always pre-sented feasible flight path waypoints or routes; how-ever, other route recommendations might have vio-lated specified flight constraints. Therefore, under eachMOA, pilots needed to maintain their flight situationawareness and carefully review the automation recom-mendations for acceptability in terms of flight con-straints. (We say more about the constraints later.)

The CDU was considered to be the lowest level of au-tomation presenting pilots with all possible route way-points for selection. Most combinations did not meetthe flight scenario constraints, including the level offuel consumption and weather to be avoided. After se-lecting waypoints, a pilot had to manually calculate fuelconsumption to determine whether the aircraft wouldhave the required amount of fuel when crossing the ini-tial approach fix (IAF). Finally, the pilot had to manu-ally program the new route into the CDU/FMS. That is,the automation assisted pilots only in terms of identi-fying the airspace and available waypoints and imple-menting the flight plan. The information-processingfunctions of generating and selecting a route were leftto the pilot. The CUI of the CDA interface presentedthe confirmed waypoints on the selected route. Figure 3shows the CDA and CDU/FMS interfaces for the CDUMOA.

The CDU+ MOA was considered to be an inter-mediate level of automation in which the CDA toolpresented pilots with a choice of several alternative

routes (instead of waypoints). Pilots selected one ofthe alternatives but were not required to calculate fuelconsumption. The DG as part of the CDA interfacepresented the fuel consumption rates for route seg-ments; however, pilots had to verify the level at theIAF when selecting a route. When the pilot decidedon a route, he or she had to “load” the route into theCDU/FMS using the “RTE” (route) page of the CDU.That is, the automation supported pilots in perceiv-ing the airspace and waypoints, generating alternativeroutes, and implementing the flight plan. The func-tion of route selection was left to the pilot, as in theCDU mode. Figure 4 shows the CDA and CDU/FMSinterfaces for the CDU+ MOA. In Figure 4a, the NDpresents all the alternative route information. Figure4b shows the “RTE” page of the CDU in which a pilotclicked the “SUGGEST RTE” option in the interface.

The CDA MOA was considered to be the highest levelof automation and presented pilots with the best routeto avoid severe weather and to meet the fuel constraint.This mode provided fully automated reroute capabilityas well as automatic loading of route waypoints into theCDU/FMS. When a pilot confirmed a suggested route,the CDA tool sent the route information to the FMS.The pilot was only required to click the “EXEC” (Ex-ecute) button on the CDU/FMS interface. That is, theautomation supported pilots in monitoring the stateof the airspace, generating flight path alternatives, se-lecting an optimal path alternative, and implementingthe flight path. Pilots were only required to verify theactions of the automation. Figure 5 shows the CDA andCDU/FMS interfaces for the CDA MOA. In Figure 5a,

Figure 3 CDA and CDU/FMS interfaces under the CDU MOA.



Figure 4 CDA and CDU/FMS interfaces under the CDU+ MOA.

Figure 5 CDA and CDU/FMS interfaces under the CDA MOA.

the ND presents the suggested route, which meets allflight constraints. Figure 5b shows the “LEGS” page inthe CDU, including the “EXEC” button used to invokethe route.

Flight workload was determined based on the dis-tance between the starting position of the aircraft andthe reroute decision point. Pilots were required to com-plete the reroute before reaching the Wilson CreekVOR, which is the first fix on the arrival. For theCDU MOA, the low- and high-workload conditionsprovided 80 and 60 nm of travel, respectively. For theCDU+ MOA, the low- and high-workload conditionsprovided 27 and 20 nm of travel, respectively. For theCDA MOA, the low- and high-workload conditionsprovided 13 and 10 nm of travel, respectively. Thosedistances and times that were provided were based on

the expert pilot input from preliminary experimenttrials.

The decision phase of the flight involved pilot use ofthe CDA tool and ran from the start of the trial untilacceptance of an arrival clearance. The implementationphase involved programming the FMS and ran from thestart of pilot use of the CDU/FMS until the “EXEC”button was pressed to activate a route. Finally, totaltime to completion spanned from the beginning of atrial until the pilot pressed the “EXEC” button on theCDU/FMS.

Finally, there were three sequential blocks of tri-als (SET1, SET2, and SET3) in which all three modesof automation were presented. The order of presen-tation of the modes differed from block to block(e.g., SET1: CDU → CDU+ → CDA, SET2: CDA →



CDU+ → CDU) and was used to assess pilot learningeffects.

2.5. Dependent Variables

TTC. The decision phase time, implementation phasetime, and total time were recorded for each trial. Thisyielded three data sets for analysis.

Modified Cooper-Harper (MCH) Workload Rating.Previous research (Kaber et al., 2009) has used theNASA-TLX (Task Load Index; Hart & Staveland, 1988)as a subjective measure of pilot workload as influencedby level of cockpit automation. However, results re-vealed some marginal workload effects of MOA, andpilots were confused by ratings of six different work-load demand components (i.e., mental demand, phys-ical demand, temporal demand, performance, effort,and frustration) without direct reference to cockpit in-terfaces or aircraft controls. Although the NASA-TLXis considered to be a robust measure of subjective work-load (Moroney, Biers, & Eggemeier, 1995), the MCHworkload rating scale was used as it follows a decision-tree approach to extract pilot workload experiencesand make direct reference to cockpit interfaces. Theoriginal “Cooper Harper assessment tool” was used toevaluate physical workload associated with use of air-craft controls (Harper & Cooper, 1986). The modifiedversion was used in the present experiment to evalu-ate how well the displays supported pilot informationprocessing and the general cognitive load experiencedin the different flight segments (Cummings, Myers, &Scott, 2006).

Heart Rate (HR). To obtain an objective measureof pilot workload, pilot cardiovascular activity wasrecorded during each phase of flight. A Polar S810iHR monitoring system was used for this purpose. HRdata were recorded every 2 seconds, and an averagevalue was calculated for each phase.

After all experiment trials, pilots were instructed torelax without performing any tasks for approximately10 minutes to measure their baseline HR. (Postexper-iment baselines have been demonstrated to be morereliable than preexperiment baselines [Kaber, Perry,Segall, & Sheik-Nainar, 2007] because of subject anx-iety in advance of testing.) During data analysis, thepercent change in HR for test conditions, as com-pared to the baseline, was calculated using the followingformula:

TestHr − BaselineHR

BaselineHR∗ 100.

Trial Success/Failure. At the end of each trial, theexperimenters recorded whether the pilot completedroute replanning by the first waypoint on the arrival.

2.6. Hypotheses

CDA trials were expected to produce the lowest work-load response in terms of MCH and HR (Hypothe-sis (H)1). Whereas the CDU MOA required decisionson individual waypoints and manual programmingof the FMS, the CDA MOA only required pilots toconfirm a suggested route. With respect to TTC, theCDA MOA was expected to produce the best perfor-mance (shortest time) because it provided maximumautomation (H2). When using the CDU MOA, pi-lots were expected to take longer to implement a newroute because of manual calculations of fuel consump-tion. Last, the CDA MOA was expected to producethe highest performance in terms of task success rate(H3) because it provided the greatest assistance withinformation-processing functions (monitoring, gener-ating, and selecting an optimal route). During CDU tri-als, pilots were required to perform many information-processing functions, including independently decid-ing on the most appropriate combination of waypointsto achieve safety and fuel constraints. Consequently,this mode was also expected to reduce the task successrate because of the number of functions in which pilotscould make errors.

2.7. Design of Experiment, Participants,and Procedure

A split-plot design was followed in the experiment in-cluding: (1) the data collection block; (2) trial set; (3)MOA; (4) order of presentation of MOAs within atrial set; and (4) workload condition. The block rep-resented a complete crossing of all the settings of themain effects (MOA and workload) for four subjects.The design included three blocks in total. Each subjectperformed three sets of trials (each set included threeMOAs, CDU, CDU+, and CDA). The trial set variablerepresented the time period within a block.

Thirteen pilots were recruited for the experimentfrom several sources in the Raleigh, North Carolina,area including the local Airline Pilots Associationchapter and the Experimental Pilots Association. Aninstrument rating and flight experience in aircraftequipped with an FMS were minimum requirementsfor participation. Eight subjects had FAR (Federal Avi-ation Regulations) Part 121 (scheduled airline) ex-perience. Five subjects had FAR Part 135 (charter)



Figure 6 Average total TTC by MOA and SET.

or Part 91 (corporate) experience. Pilot age rangedfrom 25 to 72 years with an average of 51 years.Total instrument time ranged from 175 hours to20,000 hours with an average of 3,706 hours. One par-ticipant’s data were excluded from analysis, as he didnot follow the MCH rating instructions.

After a brief survey session, pilots were familiarizedwith the simulator and the experimental setup. Theytrained on all three MOAs (CDA, CDU+, and CDU)following a TA into Raleigh-Durham International Air-port (KRDU). The basic training regimen was definedby an expert Air Force transport pilot with ATP (air-line transport pilot) certification, who had conductedcheck rides for 17 years. Pilots with “glass cockpit” ex-perience (CRT-based displays and use of an FMS) wereconsidered well qualified to participate in the experi-ment; however, all were provided with training underall MOAs until they felt comfortable with performingthe TA. Subsequently, pilots were instructed on howto complete the MCH rating form, and they donnedthe HR monitor. During test trials, each participantfollowed the flight scenario under each of the three dif-ferent MOAs in the three sequential trial blocks. At theclose of each trial, the pilots were asked to complete theMCH scale. At the end of the experiment, and after thebaseline HR recording, pilots were debriefed and theyreceived a $100 honorarium.

3. RESULTSAnalyses of variance (ANOVAs) were conducted toidentify effects of MOA, trial SET, and the flight work-load (WL) condition on pilot TTC and HR withinand across phases of flight. Because the age range ofpilots in the study was substantial, an analysis of co-variance (ANCOVA) was also conducted to assess ageas a covariate with MOA and WL on TTC, HR, andMCH. Post-hoc tests were also conducted to comparethe mean responses among the levels of MOA, SET, and

WL. Finally, contingency tables were used to examinethe success rate (a binary response) and the frequencyof MCH ratings under the three MOAs.

3.1. TTC

Figure 6 presents the mean TTC values for the vari-ous settings of the significant independent variables.ANOVA results revealed a significant effect of MOA(F (2,64) = 454.15, p < .0001) on total TTC. Post-hoc tests using Tukey’s tests indicated the CDU MOAproduced the longest TTC, whereas the CDA MOA re-quired the shortest TTC. There was also a significant ef-fect of SET (F (2,64) = 21.27, p < .0001) on pilot learn-ing. Post-hoc tests indicated that the TTC for SET1 wassignificantly longer than the TTCs for SET2 and SET3.Simple effects analysis revealed total TTC to signifi-cantly vary among sets under the CDU MOA (F (2,33)= 5.808, p = .007). Tukey’s tests showed that SET1took longer than SET2 and SET3, whereas there was nodifference between SET2 and SET3. Total TTC also sig-nificantly differed among sets under the CDU+ MOA(F (2,33) = 4.124, p = .025). Tukey’s tests indicatedthat SET1 took longer than SET3, whereas there was nodifference between SET1 and SET2 or SET2 and SET3.With respect to the CDA MOA, there was no significantinteraction with SET. There was no significant main ef-fect of the WL manipulation on total TTC (or decisionor implementation phase TTC) across all trial sets.

The decision phase TTC also revealed a significanteffect of MOA (F (2,64) = 200.96, p < .0001) on pilotroute planning efficiency using the CDA tool. Tukey’stests indicated that the CDU MOA required more de-cision time than the CDU+ and CDA MOAs. Therewas also a significant effect of SET (F (2,64) = 18.80,p < .0001) on decision time. Tukey’s tests indicatedthat there was pilot learning from SET1 to SET2 andSET3, but there was no difference in decision time be-tween SET2 and SET3.



There was a highly significant effect of MOA(F (2,64) = 1139.07, p < 0.0001) on pilot time to im-plementation of a reroute. Tukey’s tests indicated thatthe CDU MOA required the longest time, whereas theCDA MOA took the shortest time to implement. Thisparticular finding was expected in that task time differ-ences among the MOAs could, to some extent, be at-tributed to the nature of the automation interface withthe CDU requiring more button presses and mouseclicks than the CDA mode. (The results on successrate [presented later in this article] reveal differencesin pilot ability to achieve a reroute when consideringenhanced automation capability and increased inter-face usability, as assessed by the expert pilot in advanceof test trials.) There was also a significant effect of SET(F (2,64) = 9.23, p = .0003), with trials in SET1 havinga longer implementation phase than those in SET2 andSET3, according to Tukey’s test.

Regarding the results of the ANCOVA, there was nosignificant effect of age (F (1,7) = 3.36, p = .1094) andtotal instrument time (F (1,7) = 0.07, p = .7993) onpilot performance. The results of the ANCOVA wereconsistent with those of the ANOVA.

3.2. HR

ANOVA results revealed a significant effect of MOA(F (2,63) = 6.33, p = .0031) on pilot HR (seeFigure 7a). Post-hoc tests using Duncan’s method in-dicated that the CDU MOA produced a higher work-load response (increase in HR over baseline) than theCDA MOA. There was also a significant effect of SET(F (2,63) = 6.18, p < .0035) on cardiovascular ac-tivity. Duncan’s tests indicated that pilots experiencedgreater arousal during SET1 than SET2 and SET3 (seeFigure 7b). Pilots likely experienced some anxiety inadapting to the experiment conditions. In general,there was no significant effect of the WL manipula-tions on the cardiac response for any time windowacross all trial sets. Related to this finding, under some

experimental circumstances, a single process-orientedmeasure of workload, such as HR, may not be sensitiveor accurate in revealing automation condition manipu-lations. Rye and Myung (2004) offered that a battery ofobjective and subjective measures is necessary to assesscognitive workload in multitask conditions where au-tomation is applied. (The results of performance-basedand subjective measures of the workload manipulationare reported in the subsequent sections.)

During the decision phase of test trials, there was asignificant effect of MOA (F (2,63) = 7.04, p = .0017)on pilot HR. Duncan’s tests indicated that the CDUMOA produced a greater increase in HR compared tothe CDA MOA. There was also a significant effect ofSET (F (2,63) = 7.23, p = .0015). Duncan’s tests indi-cated that the decision phase HR for SET1 was differentfrom that for SET2 and SET3. However, there were nosignificant effects of MOA or SET in the implementa-tion phase of flight.

ANCOVA. There was no significant effect of age(F (1,7) = 0.43, p = .5322) and total instrument time(F (1,7) = 0.16, p = .7048) on pilot performance acrossMOAs. The results of the ANCOVA were consistentwith those of the ANOVA.

3.3. Success/Failure of Trials

An additional contingency table analysis was con-ducted on the success/failure response. This measurewas important because it essentially identified the ef-fectiveness of the MOA for supporting pilot decisionmaking in rerouting, while accounting for differencesin usability among the interfaces as perceived by the ex-pert pilot and the potential for pilots to achieve reducedplanning and implementation times. Results revealeda significant effect of the MOA on success/failure rates(χ2 = 21.70, p < .0001) under the high task WL con-dition (see Figure 8b). This was not the case for thelow WL condition. As can be seen in Figure 8b, therewas no difference in success rates between the CDU and

Figure 7 Total %HR by MOA and SET.



Figure 8 Number of successes/failures within MOA and among WL settings.

Figure 9 MCH frequency of ratings.

CDU+ MOAs; however, both of these MOAs producedmore successes than the CDA MOA. There was also asignificant effect of the WL manipulation (χ2 = 7.368,p = .007) under the CDA MOA condition. As shown inFigure 8a, the low WL condition produced higher suc-cess rates than the high WL condition. Therefore, theoutcome measure reflecting overall pilot performancewas sensitive to the workload manipulation.

In general, when the pilots had more time available(lower temporal demand) for the decision under theCDU and CDU+ MOAs and for execution, the successrate was higher than under the CDA MOA, when lesstime was available (greater temporal demand). Thatsaid, reduced task times were allotted for the CDAmode because of the level of automation and the refinedautomation interface. These results reflect the impor-tance of using a battery of measures to reveal work-load effects that may be present in pilot–automationinteraction.

3.4. MCH Ratings

The subjective workload rating data were analyzed us-ing a contingency table by counting the frequency ofratings (0–9) under each MOA. Fisher’s exact test wasused to analyze the contingency table results because

of the relatively small sample size. Test results revealedmean pilot perceived workload to significantly differamong all MOAs (i.e., CDU vs. CDU+, p < .001;CDU vs. CDA, p < .001; and CDU+ vs. CDA, p =.001). These findings were in line with the findings onthe HR response. Figure 9 presents the frequency ofMCH ratings across MCH scores (1–9). Like the HRmeasure, the MCH ratings also appeared insensitive tothe workload (task temporal demand) manipulation.It is possible that the MCH measure was less sensitivethan the performance-based assessment of workloadbecause of pilot recall bias regarding the timing ofevents in the simulation. This issue is common withsubjective measures of workload. However, the lack ofsignificance of the physiological and subjective assess-ments of the workload manipulation did not negatethe effect revealed through pilot performance.

4. DISCUSSION AND CONCLUSIONSWith respect to the hypotheses on pilot workload andperformance under the three different forms of cock-pit automation, the use of low-level automation (theCDU) led to significantly higher pilot workload (HRand MCH) and longer TTCs when compared with theCDU+ and CDA modes. These results supported our



first and second hypotheses (H1 and H2) and suggestedthat the CDU MOA supported lower situation aware-ness than the CDU+ and CDA modes with automatedand optimal route generation, respectively.

Contrary to our third hypothesis (H3), the use ofhigh-level automation (the CDA tool) led to the lowestsuccess rate (correct route selections before the startof the arrival), whereas the use of intermediate-levelautomation (CDU+) led to the highest success rate.This result was likely due to an overly rigid success cri-terion defined for the CDA mode as compared withthe CDU and CDU+ modes. The CDA mode trialsallotted pilots a disproportionately short time (13 nmand 10 nm under low- and high-workload conditions,respectively) to accomplish the decision task, as com-pared with the CDU+ mode (27 nm and 20 nm un-der low- and high-workload conditions, respectively).Here, it is important to note that the criterion distancesused to represent the specific workload settings undereach MOA were set based on the judgment of an expertmilitary transport pilot, who had some experience inusing the CDA tool. The pilot’s assessment of what waspossible with the new MOA likely exceeded the capabil-ity of the line pilots who participated in the experimentand had limited training on the technology.

The CDU and CDU+ modes of automation revealedlearning effects through the early test trials, in termsof TTC. The CDA mode, however, did not reveal asignificant learning effect. This finding suggested thatachieving SA and performance with the low and in-termediate MOAs in a reroute scenario took a longertime than when using the new CDA MOA. Further-more, the results suggested that pilots might be able toquickly adapt to the CDA form of automation for flightreplanning tasks in NextGen operations.

There was no significant effect of the starting posi-tion (flight workload) manipulation on pilot TTC orworkload responses (HR and MCH) across all modesof automation. In general, results indicated that themodes of cockpit automation influenced pilot perfor-mance and workload responses according to the hy-potheses. Some caveats of the investigation include theuse of a PC-based flight simulator. Use of a higher fi-delity, motion-based flight simulator might promotethe realism of the scenario for pilots, increase the senseof urgency in the reroute decision, and reduce pilotTTC. A motion-based simulator would also providekinesthetic cues on aircraft states, an additional infor-mation source for pilot decision making, potentiallyreducing cognitive load. In general, there is a need

for experiments, such as the present, to achieve high-fidelity simulations relative to actual flight operations,including accurately simulating aircraft responses tocontrol inputs and external forces, to promote the gen-eralizability of results for system design purposes.

Another limitation of the study was that pilot SAwas inferred based on performance with the variousmodes of automation. Other studies in the driving do-main have used such an approach for analyzing driverSA (Ward, 2000). Although performance measures areeasily integrated with simulator technologies, and pro-vide objectivity and unobtrusiveness in assessing pilotSA, they are limited in terms of diagnosticity and sen-sitivity. Factors beyond pilot SA may impact perfor-mance. Pilots may also simply get lucky in performingflight tasks successfully without achieving the neces-sary level of SA. Future research should investigatethe use of a direct measure of SA for assessing pilotperception, comprehension, and projection of aircraftstates under various MOAs, such as real-time SA probes(Jones & Endsley, 2004) but ensure that any measure-ment technique is not disruptive to performance un-der time-constrained NextGen concepts of operation.This study provides new knowledge on the relation-ship of cockpit automation and interface features withpilot performance, cognitive workload, and SA in anovel NextGen-style flight concept of operation. Theresults of this research may be used as a basis for devel-oping cockpit automation interface design guidelinesand models of pilot cognitive behavior to predict per-formance and workload with other forms of futuristicautomation, like the new CDA tool, as part of the air-craft systems design process.

ACKNOWLEDGMENTSThis research was supported by NASA Ames ResearchCenter under Grant No. NNH06ZNH001. Mike Fearywas the technical monitor. A team from APTIMA Cor-poration, led by Paul Picciano, developed the CDA toolprototypes used for this experiment. The opinions andconclusions expressed in this article are those of the au-thors and do not necessarily reflect the views of NASA.This research was completed while the fourth author,Sang-Hwan Kim, worked as research assistant at NorthCarolina State University.

References

Boeing (2008). Statistical summary of commercial jet air-plane accidents. Available at: http://www.boeing.com



Chen, T. L., & Pritchett, A. R. (2001). Development andevaluation of a cockpit decision-aid for emergency tra-jectory generation. Journal of Aircraft, 38(5), 935–943.

Coppenbarger, R., Mead, R., & Sweet, D. (2007). Fieldevaluation of the tailored arrivals concept for datalink-enabled continuous descent approach. In Proceedingsof the 7th American Institute of Aeronautics and Astro-nautics Conference (ATIO). Belfast, Northern Ireland.

Cummings, M. L., Myers, K., & Scott, S. D. (2006). Mod-ified Cooper Harper evaluation tool for unmanned ve-hicle displays. In Proceedings of the Unmanned VehicleSystems Canada Conference. Montebello, Canada.

Dao, A., Lachter, J., Battiste, V., Brandt, S. L., Vu, K. P.,Strybel, T. Z., et al. (2010). Automated spacing supporttools for interval management operations during con-tinuous descent approaches. In Proceedings of the 54thHuman Factors and Ergonomics Society (HFES) (pp.21–25). Santa Monica, CA: HFES.

Endsley, M. R., & Kaber, D. B. (1999). Level of automa-tion effects on performance, situation awareness andworkload in a dynamic control task. Ergonomics, 42,462–492.

Endsley, M. R., & Strauch, B. (1997). Automation andsituation awareness: The accident at Cali, Colombia. InR. S. Jensen & L. Rakovan (Eds.), Proceedings of the 9thInternational Symposium on Aviation Psychology (pp.877–881). Columbus, OH: Ohio State University.

Harper, R. P., & Cooper, G. E. (1986). Handling qualitiesand pilot evaluation. Journal of Guidance, Control, andDynamics, 9(5), 515–529.

Hart, S. G., & Staveland, L. E. (1988). Development ofNASA-TLX (Task Load Index): Results of empirical andtheoretical research. In P. A. Hancock & N. Meshkati(Eds.), Human Mental Workload (pp. 5–39). New York:Elsevier.

Jones, D. G., & Endsley, M. R. (2004). Use of real-timeprobes for measuring situation awareness. InternationalJournal of Aviation Psychology, 14(4), 343–367.

Kaber, D., Perry, C., Segall, N., & Sheik-Nainar, M. (2007).Workload state classification with automation duringsimulated air traffic control. International Journal ofAviation Psychology, 17(4), 371–390.

Kaber, D. B., Gil, G. H., Kaufmann, K., Kim, S. H., Veil,T., & Picciano, P. (2009). Effects of modes of cockpit

automation on pilot performance and workload in ahigh workload approach and landing scenario. Posterpresentation at the 15th International Symposium onAviation Psychology (CD-ROM). Dayton, OH: WrightState University.

Kaber, D. B., Riley, J. M., Lampton, D., & Endsley, M.R. (2005). Measuring situation awareness in a virtualurban environment for dismounted infantry training.In Proceedings of the 11th International Conferenceon Human-Computer Interaction (Vol. 9, “Advances inVirtual Environments Technology: Musings on Design,Evaluation, & Applications”). Mahwah, NJ: LawrenceErlbaum & Associates.

Kalambi, V. V., Pritchett, A. R., Bruneau, D. P. K., Ends-ley, M. R., & Kaber, D. B. (2007). In-flight planning andintelligent pilot aids for emergencies and non-nominalflight conditions using automatically generated flightplans. In Proceedings of the Human Factors and Er-gonomics Society 51st Annual Meeting (CD-ROM).Santa Monica, CA: Human Factors & Ergonomics So-ciety.

Moroney, W. F., Biers, D. W., & Eggemeier, F. T. (1995).Some measurement and methodological considerationsin the application of subjective workload and measure-ment techniques. International Journal of Aviation Psy-chology, 5, 87–106.

Rye, K., & Myung, R. (2004). Evaluation of mental work-load with a combined measure based on physiologicalindices during a dual task of tracking and mental arith-metic. International Journal of Industrial Ergonomics,35, 991–1009.

Sarter, N. B., & Woods, D. D. (1995). How in the world didwe ever get into that mode? Mode error and awarenessin supervisory control. Human Factors, 37(1), 5–19.

Ward, N. J. (2000). Automation of task processes: Anexample of intelligent transportation systems. HumanFactors and Ergonomics in Manufacturing, 10(4), 395–408.

Wright, M. C., Kaber, D. B., & Endsley, M. R. (2003). Per-formance and situation awareness effects of levels of au-tomation in an advanced commercial aircraft flight sim-ulation. In Proceedings of the 12th International Sym-posium on Aviation Psychology. Dayton, OH: WrightState University, 1277–1282.


effects of modes of cockpit automation on pilot performance and

Documents