lable at ScienceDirect

Applied Ergonomics 42 (2011) 348e357

Contents lists avai

Applied Ergonomics

journal homepage: www.elsevier .com/locate/apergo

Effect of cognitive load on speech prosody in aviation: Evidence from militarysimulator flights

Kerttu Huttunen a,*, Heikki Keränen a,b, Eero Väyrynen b, Rauno Pääkkönen c, Tuomo Leino a,d

a Institute of Clinical Medicine/Department of Otorhinolaryngology, University of Oulu, P.O. Box 5000, FI-90014 Oulun yliopisto, FinlandbMediaTeam Oulu, Department of Electrical and Information Engineering, University of Oulu, P.O. Box 4500, FI-90014 Oulun yliopisto, Finlandc Finnish Institute of Occupational Health, P.O. Box 486, FI-33101 Tampere, FinlanddAir Force Command Finland, P.O. Box 30, FI-41161 Tikkakoski, Finland

a r t i c l e i n f o

Article history:Received 5 April 2010Accepted 17 August 2010

Keywords:PilotVoiceWorkload

* Corresponding author. Tel.: þ358 50 511 4947; faE-mail addresses: kerttu.huttunen@oulu.fi (K. Hu

oulu.fi (H. Keränen), eero.vayrynen@ee.oulu.fi (E. Vttl.fi (R. Pääkkönen), tuomo.leino@mil.fi (T. Leino).

0003-6870/$ e see front matter � 2010 Elsevier Ltddoi:10.1016/j.apergo.2010.08.005

a b s t r a c t

Mental overload directly affects safety in aviation and needs to be alleviated. Speech recordings areobtained non-invasively and as such are feasible for monitoring cognitive load. We recorded speech of 13military pilots while they were performing a simulator task. Three types of cognitive load (load onsituation awareness, information processing and decision making) were rated by a flight instructorseparately for each flight phase and participant. As a function of increased cognitive load, the meanutterance-level fundamental frequency (F0) increased, on average, by 7 Hz and the mean vocal intensityincreased by 1 dB. In the most intensive simulator flight phases, mean F0 increased by 12 Hz and meanintensity, by 1.5 dB. At the same time, the mean F0 range decreased by 5 Hz, on average. Our resultsshowed that prosodic features of speech can be used to monitor speaker state and support pilot trainingin a simulator environment.

� 2010 Elsevier Ltd and The Ergonomics Society. All rights reserved.

1. Introduction

Military aviation and military pilots represent a considerableproportion of the traffic volume and staff in aviation. Despitetechnological advances in humanemachine interaction, spokencommunication still plays a central role in aviation safety andmission effectiveness in military aviation. Speech is, therefore, animportant research area also within this branch of aviation.

Cognitive load of individual pilots and crew is present on allflights. It refers to pressure on human central information pro-cessing in the domains of perception, memory, logical reasoningand learning. In research, cognitive load is most often measured incircumstances where participants simultaneously monitor severalstimuli or perform mathematical, linguistic or psychomotor tasks.Hansen et al. (2000) definedworkload as a subject-dependent levelof used capacity of resources. Workload (of which cognitive load ispart) causes stress (the body’s response to load), and stresssymptoms can be detected in speech and voice. Changes in speechdue to workload/cognitive load or stress-induced speech changes

x: þ358 8 315 3459.ttunen), hkera@mail.student.äyrynen), rauno.paakkonen@

and The Ergonomics Society. All ri

are often difficult to differentiate from each other. This is under-standable, as, e.g., Hart and Hauser (1987) found pilots’ in-flightself-ratings of workload and stress level to be highly correlated(correlation coefficient 0.94).

Prosody (rhythm, stress and intonation of speech) conveys manyimportant features of spoken messages. Knowing how in-flightprosodic features change is useful in aviation. The most importantaspect is safety: air traffic controllers (and in military aviation, alsofighter control officers) need to recognize changes in the mental orphysical state of pilots which prosody gives information on. Addi-tionally, prosody can be used to improve speech intelligibility. Whenindividual changes in prosody in stressful situations are known, thisinformation can be used to teach compensative speech mechanismsto pilots so that communicative effectiveness can be improved (Livelyet al.,1993). Pilots alsoneed to learnhowtouse their vocal intensity tooptimize the use of a radio communication system based on ampli-tude modulation in situations where several people transmit on thesame radio frequency band simultaneously or almost simultaneously.Furthermore, as prosody helps in differentiating speakers from eachother, it enhances formation and maintaining of situation awarenessin the hectic work assignments of military pilots. It is, therefore,important to learn the nature and amount of changes that differentfactors (e.g., cognitive load, emotional stress) have onprosodyat bothgroup and individual levels. As speech samples can be obtained non-

ghts reserved.

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357 349

invasively, features of speech can serve as practical and safelyobtainable indicators of cognitive load and stress.

1.1. Cognitive load in military aviation

In multi-task flight environments, where information fromseveral senses must be perceived and rapidly integrated intoplanning of actions, loads of situation awareness (SA), informationprocessing and decision making are simultaneously present. SArefers to perception of the environment critical to decision makingin complex, dynamic situations (Endsley, 1995, 1999). It is related toperceiving, understanding, projecting, updating and assessinginformation available. In the cockpit of a fighter aircraft or a flightsimulator of a fighter aircraft, the pilot is provided with an endlessstream of data on, e.g., the flight task, aircraft and other aviators.This information needs to be continuously monitored, perceived,quickly processed, prioritized and used to make decisions onactions related to aviating, navigating and communicating toperform the tasks set for the mission. Processing of massiveamounts of information challenges attention, perception, long termand working memory, selection of actions and decision making(Svensson et al., 1997). Rapid changes in environment have beenfound to cause difficulties to maintain SA (Svensson and Wilson,2002).

When too low or too high, cognitive load increases the risk offlight accidents. Sustained operations like transcontinental flightschallenge the vigilance and other parts of cognitive capacity ofaviators and pose a clear risk of mishaps (Armentrout et al., 2006).However, high levels of cognitive overload are especially related tosituations in which abrupt bursts of a large amount of informationneed to be processed quickly. Therefore, high cognitive load amongaviators, among others, associated with strong time pressure,serves as a noticeable stressor and causes psychophysiologicalstress reactions like an increase in heart rate and decrease in heartrate variation, and increased higher blood pressure, muscle tension,perspiration, anxiety and fatigue (Lee and Liu, 2003). Anxiety, stressand fatigue impair particularlyworkingmemory functions of a pilot(Stokes and Kite, 2003, p. 59). Take-off, approach and landing aretypical flight phases of normal high cognitive load in civil aviation(Lee and Liu, 2003; Silberstein and Dietrich, 2003; Wilson, 2002),whereas the intense phases of combat exercises and sudden systemdegradations represent phases of extraordinary high cognitive loadfor military pilots (Svensson andWilson, 2002). Military pilots mayalso have to fly in formation, use night vision devices to fly at night,operate in remote, unsupported airstrips, perform low-level flights,and drop paratroops and cargo (Skinner and Simpson, 2002).

High time pressure, task criticality and mission complexityincrease military pilot’s workload, and affect SA and pilot perfor-mance (Svensson et al., 1997; Svensson and Wilson, 2002). Whenhigh information load exceeds pilot’s compensation capacity, itresults in impaired flight performance (Svensson and Wilson,2002). Svensson et al. (1997) found that simulator flight perfor-mance of military fighters deteriorated as a function of increasedinformation load. The pilots studied, e.g., deviated from prescribedaltitude and missed targets to be detected as the task complexityincreased. Technological and procedural solutions have beensought to reduce pilot workload (e.g., Casner, 2009; Elmenhorstet al., 2009).

Flying requires group interaction that is realized by operatingthe aircraft and communicating with air traffic controllers, fightercontrol officers, and other aviators. High cognitive load affectsnegatively speech and communication of pilots (Skinner andSimpson, 2002), sometimes with drastic consequences. Workoverload has also been shown to impair a pilot’s ability to shareessential information with the crew members (Silberstein and

Dietrich, 2003). Communication tasks themselves required inaviation also provide a significant source of cognitive load (Hart andHauser, 1987). Because the majority of tasks in aviation involvecommunication, there is a need to explore effects of cognitive loadon communication and speech.

1.2. Prosody and aviation

Prosody has an important role in communication because itsupports and supplements information that segmental featuresconvey from speech. Of the prosodic features, the speech rate usedis often higher in military aviation than in civil aviation. Typical toaviation, standard radiotelephony phraseology specially con-structed for the exchange of messages is used in most parts offlights. Its purpose is to avoid misunderstandings in communica-tion. In addition, tactical phraseologies are used inmilitary aviation.Due tomany speakers using the same radio frequency band and thehigh amount of information to be transmitted within a short timespan, military pilots are instructed to communicate briefly andeffectively.

Information on speech features can be used to find out andmonitor speaker state in stressful situations. For example, Saitoet al. (1980) analyzed cockpit voice recorder data and found thathypoxia had been the probable cause of a fatal accident theystudied. Applications of prosody research can be used, for example,in improving speech synthesis and automatic speech recognition(Batliner et al., 2003; Clavel et al., 2008; Fernandez and Pickard,2003; Jones and Jonsson, 2008). Automated speech and speakerrecognition systems will be integral parts of future aircraft and airtraffic control systems. For example, in addition to the use of callsigns in aviation, aircraft identification systems are under devel-opment to enhance safety by extracting data from pilots’ voices.Automatic emotion recognition has been investigated but is stilla relatively new research area (see, e.g., Truong et al., 2007). It is,therefore, important to explore features of speech under stress andhigh cognitive load with traditional acousticephonetic measure-ments. Prosody is not as vulnerable to background noise as aremany segmental features of speech (Grant and Walden, 1996).Additionally, Brockx and Nooteboom (1982) found in their studythat prosodic information can help a listener, e.g., in auditoryfigure-ground perception: a listener can use prosodic cues tosegregate one voice from another amidst background noise. This isone of the valuable uses of prosodic information inmilitary aviationradio communication, where multiple speakers often occupy thesame frequency band.

1.3. Prosodic features as indicators of cognitive load andpsychophysiological stress

Many acoustic parameters like fundamental frequency (F0),intensity, energy attack and decay gradients, jitter, physiologicalmicrotremor, duration of pauses and speech segments, speech rate,speech disfluencies, formant locations, and spectral tilt (high-energy frequency ratio of speech) have been studied as task load orworkload-induced vocal features of psychological stress (Johnstoneand Scherer, 2004; Rothkrantz et al., 2004; Scherer et al., 2002).

Speech under stress or a high workload is often characterized bya faster rate of speech (Johnstone and Scherer, 2004; Silberstein andDietrich, 2003). However, findings of no change and a slower rate ofspeech have also been reported (Rothkrantz et al., 2004). A fasterspeech rate is often associated with less intelligible speech, becauseaccurate articulation requires good co-ordination between dozensof muscles and is, therefore, difficult to maintain under high timepressure. F0 is a powerful way to express emotions and linguisticelements like word stress and sentence stress (Hirschberg, 2002)

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357350

and it also affects clarity of speech. Speech intelligibility is alsoparticularly enhanced by increased intensity of speech and varia-tion in it (Lively et al., 1993). In the workload research literature,high F0 and intensity of voice have been the speech parametersmost often associated with high cognitive load. Changes in speechdue to high load have been found to be very consistent withina single speaker, but variable between different speakers. Becauseof this finding, further research in this area is warranted.

Taken together, the prosodic features of speech are found to besensitive to stress andmental overload (Hagmüller et al., 2006). It isimportant to study prosody in aviation to detect cognitive/mentaloverload on the basis of a pilot’s speech, and thereby obtaininformation that can be used to improve flight safety. Informationon features of speech can be used especially in pilot training, e.g., tofacilitate pilots’ self-monitoring in order to improve their speechintelligibility in radio speech communication. This information isalso needed in the development of technology for humanemachineinteraction (voice-operated systems). As laboratory studies andfield studies have been found to be comparable in measuringdecrements in cognitive functions (Lieberman et al., 2006;Svensson and Wilson, 2002; Ylönen et al., 1997), it is feasible tostudy the effects of cognitive load on speech features in simulatorenvironments without interfering physical factors. In aviation,practicing in simulators is part of pilots’ normal working environ-ment, and studies conducted in simulators often reflect theirworkload very realistically.

The aim of the present study was first to find out how certainprosodic features change in different simulator flight phases andsecondly, how they change as a function of various degrees of threetypes of cognitive load (situation awareness, perceiving and pro-cessing of information and decision making during the flight).Thirdly, we also wanted to get an insight into the role of certainbackground factors of thepilots on changes in theprosodic features ofspeech.

2. Methods

Before the data collection, the study protocol was approved bythe Ethical Committee of the Northern Ostrobothnia HospitalDistrict, and all the participants signed a written consent.

2.1. Experimental design

Each pilot performed one simulated combat flight in an F/A-18Hornet WTT (Weapons Tactics Trainer) flight simulator. Speech ofthe participants was recorded during the flight, and basic prosodicfeatures were measured from each utterance of the recordings andused as dependent variables in the study. Three different types ofcognitive load assessed after the flight served as independentvariables.

2.2. Participants

Altogether 15 qualified male Hornet fighter pilots, active inmilitary flight service, participated in a demanding simulator flight.Unfortunately, speech data could not be obtained from oneparticipant and cognitive load evaluation from another participantbecause of an unexpected failure in audio or video recordings.Hence, data from 13 pilots were available for analysis. Mean heightof the participants was 178 cm (range 171e183 cm), mean weight78 kg (range 71e87 kg), and mean age 28 yrs (range 25e34 yrs).Concerning living habits that affect voice, altogether two of the 13pilots smoked, three used snuff and one did both. Ten (77%) of thepilots drank coffee (mean 3 cups, range 1e7) in themorning of theirsimulator test day.

2.3. Noise and speech recordings

In a DOME-WTT flight simulator, a cockpit is placed within a ballwith a diameter of 12 m. The cockpit itself isfixed inplace, and hencedoes not cause noise itself, but some noise is generated by the videoprojectors that project the virtual flight and combat scene on onehemisphere of the ball. The noise of the electric motors of the videoprojectors gets louder during more intensive flight phases as simu-lated battle scenes change rapidly. Background noise measurementswere, therefore, carried out within the WTT flight simulator cockpitwith a LarsonDavies Spark 706 noise dosimeter. The noisemeterwasplaced in the cockpit above the seat belt level of the pilot. Backgroundnoise in the cockpit was recorded and averaged as A-weightedequivalent sound pressure levels (Leq(A)) over one-minute segments.Instead of averaging over each flight phase or the entire flight, aver-aging over one-minute segments was done to achieve more fine-grained information on the existing background noise level at thetime the pilots produced each utterance. In addition to videoprojectors, background noise possibly affecting the pilot’s speechproduction is also generated by radio communication. The volume ofthe simulator radio was adjusted freely by the pilots. However, itslevel was not measured from the audio recordings.

Pre-amplifiers were constructed for each of the Sennheiser KE4-211 electret miniature microphones used in the tests according tothe instructions of the microphone manufacturer. The Sennheisermicrophones had a flat frequency response across the frequenciestypical to human speech. In the flight simulator, each pilot had oneSennheiser microphone (Sennheiser KE4-211-1) attached underthe flight helmet near the tragus at the entrance of the ear canal,one inside the oxygen mask (Sennheiser KE4-211-9 with anomnidirectional directivity pattern; it was not possible to usea cover to reduce wind noise) and one on the shoulder.

Before the measurements, the sound recording systems werecalibratedwith aBrüel&Kjær4231 sound level calibrator. A referencesignal of 94 dB (at 1000 Hz, 0 dB re 20 mPa) was recorded at thebeginning of all the Sony DAT tapes used. During the flights, fourchannels of sounds were recorded simultaneously from each pilot,with a sampling rate of 44.1 kHz and 16-bit depth, with altogethertwo dual-channel Sony TCD-D7 or TCD-D8 DAT (digital audio tape)recorders. Additionally, the speech of the fighter control officer wasrecorded stereophonically in the combat centre. After the recordings,the data on the DAT tapes were transferred to CD-ROMs. The speechdata were subsequently filtered with a 55 Hz high-pass filter toeliminate any low-frequency artefacts caused by mains interference.

2.4. Procedure

After seating, the pilots put on their individually fitted GentexACS fighter aircrew helmets with an oxygenmask. Because the testswere run in a flight simulator environment, no extra hearingprotection was used in addition to that provided by the earmuffsand insulation of the flight helmet. After seating and a five-minuterest, the study design was counterbalanced across the incomingparticipant’s identification number: every other pilot started thetests with the simulator flight and the rest of the pilots started witha control phase consisting of a 20-min reading task. In this controltask, the participants and their fighter control officer read in dia-logue transcript from a real fighter flight. The study design selectedwas necessary as heart rate was also monitored during the simu-lator flights, and it was important to establish a baseline for theheart rate measurements to find out what kind of effect speakingalone had on heart rate. However, more detailed results on theheart rate results are out of the scope of the present paper.

The task of the participants was to serve as the leader of a pair ofF/A-18 Hornets. Two flight instructors alternated in serving as

Table 1Definitions of three different cognitive loads (situation awareness load, information load and decision load) with their references in the research literature, as presented to theflight instructor assessing the amount of these cognitive loads in each flight phase.

Situation awareness load (SA LOAD) Information load (INF LOAD) Decision load (DEC LOAD)

(Endsley, 1993, 1995, 2000; Flach, 1995; Wickens, 2002) (Berthold & Jameson, 1999; Endsley, 1995;Mendoza and Carballo, 1998;Svensson et al., 1997)

(Brenner et al., 1994; Hutchins et al., 1996;Svensson et al., 1997)

Perception of the elements in the environment (level 1):location, altitude, heading (of one’s own and other aircraft),target detections, aircraft system status (e.g., warning lights,radar limitations, fuel level), ground threat and groundobstacle locations, etc.

Amount of information requiring simultaneousattention and memory (all information producedby the simulator system and received viaradio communication)

Time available for decision making

Comprehension and integration of information (level 2):timing and status of the flight, impacts of systemdegrades/malfunctions, time and distance possible onfuel available, tactical posture of enemy aircraft(offensive/defensive/neutral), etc.

Quality of information (complexity and criticality) Criticality of the decisions neededto be made

Projection of the perceived information to the future status(level 3): planning and assessment of projected aircrafttactics and maneuvers, firing position and timing, etc.

Amount of decisions needed to be madewithin a short time span

NOTE: SA does not yet include decision making (Endsley, 1995),although it is the basic foundation of decision making(Wickens, 2002)

Decision complexity (single/multi-choicenature of decisions; are there many choicesfrom which to select)

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357 351

a wingman for the pilots. The flight phases planned beforehandwere: (1) start of the flight mission at 24,000 ft (7315 m) anda control air patrol (CAP), (2)first tacticalmaneuver, (3)first beyond-visual-range (BVR) interception of an enemy fighter, (4) secondtactical maneuver, (5) second beyond-visual-range (BVR) intercep-tion of an enemyfighter, (6) third tacticalmaneuver, (7) interceptionof an enemy attack aircraft formation, (8) break from combat, (9)return towards the base at high altitude, (10) return towards thebase at low altitude, (11) IFR (instrument flight rules) initialapproach, (12) IFR vectors and (13) ILS (instrument landing system)inminimumweather conditions and landing (formore information,see Lahtinen et al. (2007) and Hannula et al. (2008)). Communica-tion took place in Finnish during the flights. The simulator flightconstructed was considered to be cognitively very demanding evenfor experienced pilots. During the flights, the flight performance ofeach pilot was assessed by a fighter control officer (an experiencedflight instructor) on an ordinal scale of 1e5. Scores were given forthe different interception phases, firing parameters, battlegeometrics, break from combat, ILS approach and landing, forleading the pair and for radio communication. The scores of thesevariables were combined to form a total flight performance score(with an accuracy of 0.25). During the flight performance assess-ment, all the flight and simulator system data together with audioand video monitoring data on the pilot sitting in the simulatorcockpit were available to the fighter control officer. As the pilotswere informedof being assessed, this consciousness probably addedto their cognitive load. The mean number of flight phases duringeach simulator flight performed was 13, and their mean durationwas 2 min, 2 s (range 15 se7 min, 45 s). The flight (from CAP tolanding) lasted 25 min, on average (SD 2, range 24e32 min).

2.5. Measures

2.5.1. Cognitive load analysesWhen each participant performed his flight, all essential flight

information was video-recorded from the displays of the combatcentre (operations room of the simulator). These data included allcommunication between the pilot, his wingman, air trafficcontroller and the fighter control officer. Cognitive load of thepilots was assessed afterwards using these video-recorded multi-monitor flight data. The monitor data provided information for the

rater about the most important parameters of the simulator flights(heading, speed, altitude, radar information, firing parameters,fuel level etc.). The load was rated subjectively by the same flightinstructor who supervised the flight performance of the partici-pants during the simulator flights. Although the simulator flighttask constructed was identical for each participant, all the flightswere unique, as the decisions made by the pilot and the wingmanduring the different flight phases resulted in individual workloadsand flight performance. Therefore, every pilot served as his owncontrol; prosody results of each participant were compared witheach other in different flight phases, not with results of otherpilots.

The variables used in the evaluation of cognitive load weresituation awareness (SA load), information (INF load; amount,complexity and criticality of information load) and decision-makingload (DEC load). These variables were selected on the basis of rele-vant research literature (e.g., Brenner et al., 1994; Endsley, 1993;Svensson et al., 1997) and practical knowledge about a pilot’s workassignments in military aviation. No universally accepted modelsare currently available especially for assessment of situationawareness, but of the specific theoretical models available, weadopted a three-level model proposed by Endsley (1995), repre-senting a cognitive theory that uses an information processingapproach, because it is widely used. According to Endsley (1994,1995), factors like attention, working memory and other informa-tion processing mechanisms, goals and expectations of a persontogetherwithworkloadand stress are independentphenomena, butthey all affect situation awareness. To achieve sufficient validity inthe assessment, the conceptual content of the areas to be assessedwas detailed in thewritten instructions given to the flight instructorfor the post-trial assessment (see Table 1). As can be seen from thetable, our view of cognitive load is more process-oriented thanproduct-oriented, with themeasurement of cognitive load focusingon the underlying processes required to complete the simulatorflight task successfully.

The flight instructor marked down his evaluations of cognitiveload levels in each flight phase on a Visual Analogue Scale (VAS)(see e.g., McCormack et al., 1988;Wewers and Lowe,1990). VASwasa 100-mm horizontal line on which the scores of the load variablesranged from zero (at left hand end of the line, implying no cognitiveload at all) to one hundred (at right hand end of the line, implying

Fig. 1. Distribution of three cognitive load variables (load on situation awareness,

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357352

the highest imaginable level of cognitive load). Vertical marks onthe VAS were measured as a distance from the left hand end of theline, and coded as millimeters.

2.5.2. Speech analysesOf the many features of speech measured, F0 and vocal intensity

were selected as variables to be examined in detail, because theyhave been shown to be the most unanimously indicative ofcognitive load and psychological stress (e.g., Douglas-Cowie andCowie, 2000; Hagmüller et al., 2006; Prinzo and Britton, 1993;Wittels et al., 2002).

The utterances spoken by the pilots were transcribed usinga transcription tool specially constructed by our research team forthis purpose. For automatic analysis of F0 and intensity, a ceps-trum-based voiced/unvoiced segmentation and time domain F0extraction algorithm using waveform matching was employed (fordetails, see Seppänen et al., 2003). Mean F0 and mean RMS (rootmean square) intensity values were analyzed from the utterances ofthe flight phases from which the assessment of cognitive load wasalso obtained.

The reliability of the automatic analysis tool had been testedearlier against semi-manual measurements in very challengingdata obtained from real F/A-18 Hornet flights (Keränen et al., 2004).The data from the speech measurements were merged with thebackground noise and cognitive load data on the basis ofsynchronized time between the different data sources.

information processing and decision making) in the simulator flight phases as assessedby the flight instructor.

3. Results

The simulator flight recordings from CAP to landing comprised8153 words in total, distributed over 1536 utterances. Each flightphase contained 118 utterances, on average (SD 64.2, range33e207). Background noise (as LeqA) in the cockpit, averaged overone-minute intervals, was, on average, 75 dB(A) during the simu-lator flights with a SD of 1.9. The mean noise level per flight phaseranged from 73 to 75 dB(A).

During the flights, the mean situation awareness load on a VASwas 32, the mean information load was 28 and the mean decisionload was 31. The three types of cognitive load were stronglyintercorrelated (Pearson’s rho from 0.916 to 0.944, p< 0.01). Thehighest cognitive loads took place in flight phases 3e7 (Fig. 1),during which the mean scores were 51 (SA load), 44 (INF load) and49 (DEC load). In these phases, the task of the pilots was to intercepta virtual single enemy aircraft and an aircraft formation.

In our first research question, we were interested in how theutterance-level mean F0 and the mean vocal intensity of the pilotsvaried in different flight phases. The mean F0 and mean F0 changevalues of individual pilots in different flight phases are given inFig. 2, and the mean intensity and mean intensity change values, inFig. 3. During the first five utterances in flight phase 1, that is, in thebeginning of the flight, the mean F0 of the pilots was 112 Hz (SD 22,range 79e174). Although the mean F0 of most of the pilots wasbetween 100 and 120 Hz in the majority of the flight phases,individual variation was seemingly large. The lowest F0 valuessuggest the presence of vocal fry (speaking with a creaky voice) inthe speech of one or more pilots. During the first five utterances inflight phase 1, vocal intensity was 109 dB, on average (SD 3.2, range101e116). The measured values were high because the miniaturemicrophone used in the recordings was placed near the mouthinside the oxygen mask. A correlation of 0.479 (Pearson’s rho,p< 0.001) was found between the mean F0 and mean intensityvalues when all the flight phases (1e13) were taken into account.We also found that mean F0 and mean F0 range were intercorre-lated: the higher the mean F0, the larger the mean F0 range

(rho¼ 0.463, p¼ p< 0.01). Mean F0 range was also associated withmean intensity range (rho¼ 0.467, p¼ p< 0.01).

Regarding our second research question e how prosodicfeatures changed as a function of the three cognitive loads e wefound that both themean F0 (Fig. 4) and themean vocal intensity ofthe pilots (Fig. 5) increased when cognitive loads got higher. Thevariables “Mean F0 change” and “Mean intensity change” werecalculated by subtracting the mean results of each pilot in each offlight phases 1e10 from the mean of his own F0 or vocal intensityresults in the three last flight phases (mean of the mean values inphases 11e13 in which cognitive loads were low, that is, under 10).The mean F0 change was, on average, 7 Hz (SD 17.0) and the meanintensity change was 1 dB (SD 1.9 dB). In the most intensive flightphases, 3e7, the mean F0 change was even higher (12 Hz, onaverage), as was also the mean intensity change (1.5 dB). Addi-tionally, the mean F0 range of each pilot was reduced by 4 Hz, onaverage (SD 6 Hz), in flight phases 1e10, indicting a clear increase inlaryngeal muscle tone. Individual mean intensity ranges were alsoreduced (by 1 dB, on average, SD 1) during the first 10 simulatorflight phases. In the most intense flight phases, 3e7, the mean F0range decreased somewhat more (by 5 Hz, on average). However,although a decrease in F0 range was the general tendency, theaverage F0 range of four participants increased; in one pilot theaverage increase was distinctive (48 Hz). The change in prosodicfeatures was slightly different depending on the cognitive load typein question. Fig. 4, based on spline interpolation with smoothing(smoothing parameter s¼ 60), reveals that mean F0 change wassomewhat more prominent as a function of DEC and INF loads thanit was as a function of SA load. The same trend was seen in meanintensity change (Fig. 5).

In the physiology of speech production, vocal intensity affectsthe fundamental frequency, especially in loud speech. This isbecause both F0 and intensity are modified by muscle tension andsubglottal pressure. Both physiological changes can naturally be

Fig. 2. Mean F0 and mean F0 change values of individual pilots during different phases of the simulator flight. The mean F0 change was calculated by subtracting the mean results ofeach pilot in each of flight phases 1 to 10 from the mean of his own F0 results in the three last flight phases (11e13, having the lowest cognitive load). Fragmented lines are caused bymissing data that resulted from pilots occasionally skipping a few flight phases.

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357 353

caused by the same factor. This interconnection was also seen inthe present data. Mean F0 increased 4.7 Hz/dB, on average(4.5 Hz/dB without taking into account one participant whoshowed extreme values). Furthermore, mean F0 and meanintensity changes were also rather strongly and statisticallysignificantly correlated with each other (Pearson’s correlationcoefficient 0.530, p< 0.01). Mean F0 change was 7.9 Hz (median4.7 Hz) for every decibel of mean intensity change. The rangechanges were also intercorrelated: when F0 range was reduced,usually intensity range was reduced, too (Pearson’s rho¼ 0.572,p< 0.01). In addition that changes in the mean F0 and mean vocalintensity of the pilots were statistically significantly correlatedwith each other, they correlated with the three cognitive loadsmeasured (Table 2). Mean F0 range change correlated statisticallysignificantly with neither SA load nor INF load, and although its

Fig. 3. Mean intensity values and mean intensity change values of individual pilots duringsubtracting the mean results of each pilot in each of flight phases 1 to 10 from the mean ofcognitive load).

correlation with DEC load was statistically significant, the corre-lation coefficient was very low (r¼ 0.082).

In our third research question we were interested in the effectof certain background factors (physical and physiological char-acteristics and flight experience of the pilots) on the changes inthe prosodic features of speech. We especially wanted to know ifany of the background factors could serve as a confounding factor.In a visual inspection of the scatter plot and box-plot figures,no trends in the distribution of F0 change or intensity changevalues were seen as a function of the age, height or weight of theparticipants. The same applied regardless of whether theysmoked or used snuff or the number of cups of coffee they hadconsumed in the test morning. Additionally, neither flight expe-rience, flight performance in the simulator task nor cockpitbackground noise was associated with the F0 and intensity

different phases of the simulator flight. The mean intensity change was calculated byhis own vocal intensity results in the three last flight phases (11e13, having the lowest

Table 2Intercorrelations (Pearson correlation coefficients) between mean F0 and meanintensity change and range change and the three cognitive loads (assessed by theflight instructor on a VAS) in flight phases 1e10.

Mean F0range change

Mean intensityrange change

SA load INF load DEC load

Mean F0 change 0.350** 0.157** 0.321** 0.427** 0.420**Mean intensity

change�0.104** �0.125** 0.240** 0.306** 0.253**

SA load 0.916** 0.924**INF load 0.944**

Pearson’s rho (two-tailed), **¼ p< 0.01.

Fig. 4. Mean F0 change (mean change in flight phases 1 to 10 compared with the threelast flight phases, 11e13, which contained only a slight load) as a function of threedifferent cognitive loads.

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357354

changes. Careful examination of our data with the help of figuresallowed us to quite safely conclude that none of the backgroundfactors had significant effects on the mean F0 and intensitychanges.

Fig. 5. Mean intensity change (mean change in flight phases 1 to 10 compared withthe three last flight phases, 11e13, which contained only a low load) as a function ofthree different cognitive loads.

4. Discussion

4.1. Main findings

According to our main findings, the mean F0 and mean vocalintensity of our participants increased and their range decreased asa function of the three cognitive loads studied. Like an increase inmean F0, a reduced mean F0 range also indicates higher muscletone in the laryngeal muscles. However, only DEC load was statis-tically significantly associated with F0 range change (although thecorrelation coefficient was extremely small), and individual varia-tion in F0 range was large; although the F0 range of nine pilotsdecreased, therewere four pilotswhose F0 range increased. NeitherF0 nor intensity increased linearly as a function of cognitive load,and in a quarter of the pilots the prosodic features did not behave inthe same way as they did in the rest of the group. Generally,changes in prosody were steeper when the VAS scores of cognitiveload were high (above 50 or 60). In our data, the mean F0 of threepilots did not change practically at all. This interestingly raisesa question of individual patterns in responding to cognitive load,and hence inherent or learned stress coping strategies.

4.2. Prosodic changes as a function of three types of cognitive load

Considering the inherent nature of and strong intercorrelationsbetween the three cognitive load types analyzed, it is quiteunderstandable that the mean F0 of the pilots increased more asa function of INF and DEC load compared with SA load. EspeciallyDEC load probably contains accumulated features of both SA andINF loads. In stressful situations, cognitive load may also interferewith planning and execution of speech and result in changes inprosody (Johnstone and Scherer, 2004; Tolkmitt and Scherer, 1986).

4.3. Results in relation to other studies

Themagnitude of changes of F0 and vocal intensity in the speechof our participants measured in a simulator environment are situ-ated between the results of studies from authentic work environ-ments (Wittels et al., 2002) and from a laboratory (Griffin andWilliams, 1987; Lively et al., 1993; Mendoza and Carballo, 1998;Scherer et al., 2002), depending on, e.g., the nature and level ofcognitive load and speech data used or collected in the variousstudies. Lively et al. (1993) found a general decrease in F0 variability(decreased SD) in three of their five participants. Parallel to thestudies by Lively et al. (1993) and Johnstone and Scherer (1999), theutterance-level mean F0 range of the pilots we studied was some-what reduced as a function of higher decision-making load. John-ston and Scherer discussed that their finding of limited pitch rangein anxious speech resulted from general tenseness of the laryngealmusculature, which limits adjustment of vocal cord tension andlaryngeal position. Likewise, we expected the muscle tension of ourparticipants to increase during the most intensive flight phases.

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357 355

However, despite the increased muscle tension associated withincreased F0, the range of the fundamental frequency is not neces-sarily associated with perceived extreme emotional stress. Forexample, listeners in the study by Protopapas and Lieberman (1997)did not associate emotional stress with the increased F0 fluctuation(jitter) of a helicopter pilot who actually was experiencing distress.

4.4. Possible intervening and competing factors

When speech data are collected in an environment with at leastsome background noise it is important to estimate the risk of noiseserving as a competing factor because noise may cause theLombard effect to come out. The Lombard effect is an involuntarytendency of a speaker to increase vocal intensity of speech amidstbackground noise (Junqua, 1996). Because our participants usedindividually fitted flight helmets with sound protection, based oninformation derived from research reports (Kuronen, 2004; Olsen,1998; Zahorik and Kelly, 2007), we estimate the level of back-ground noise to have been at a normal conversational level (around60 to 66 dB(A) at the entrance of the ear canal of the pilot). Anyway,if the Lombard effect emerged, its effect would have been rathersimilar in the different flight phases, as the mean level of back-ground noise was rather stable (mean of the flight phase meansranged from 73 to 75 dB(A)).

Under mental pressure, both the fundamental frequency andintensity of voice may change simultaneously, but F0 may alsochange independently from vocal effort. According to the results ofJessen et al. (2005) based on 100 male speakers, F0 increases 5 Hzper decibel when a person is changing vocal effort from normal toloud speech. We noticed a clear association between mean F0change and mean intensity change (Pearson’s rho¼ 0.530); meanF0 increased by 7.9 Hz/dB. We, therefore, estimate the increase inF0 to have resulted mainly via an increase in vocal effort (e.g.,making the vocal cords adduct more rapidly), and the change inboth mean F0 and mean intensity to have resulted from cognitiveload. The change in F0 matched well with such an effect of vocaleffort, that has been found to be associated with a doubling ofcommunication distance (Traunmüller and Eriksson, 2000).

4.5. Assessment of data collection

We did not perform any reliability measurements of cognitiveload assessment, like Lively et al. (1993) did. However, the flightinstructor that assessed cognitive load after the series of tests wasexperienced and served as the fighter control officer (and henceassessed the flight performance of the pilots) during the simulatorflights. Regarding the validity of the assessment, we have recentlyfound that of physiological measures, heart rate (Hannula et al.,2008; Lahtinen et al., 2007), and heart rate variation (ReR inter-vals, Hannula et al., 2007) were also associated with cognitive loadduring the same simulator flights reported in the present paper.Because the level of cognitive load was associated with both speechfeatures and heart rate, both commonly known to be indicators ofcognitive and emotional load, we can quite safely assume that theassessment of cognitive load was valid. Simultaneous changes inspeech and cardiac autonomic regulation, therefore, serve asa cross-validation of our measurement systems.

We collected speech data in a military flight simulator becausewe wanted to focus on exploring the effects of cognitive loadwithout intervening physical factors. Simulator studies still need tobe compared with data from real flights to find out their corre-spondence. Keränen et al. (2004) and Keränen (2005) reported thatthe in-flight mean F0 values of Finnish F/A-18 Hornet pilots (N¼ 5)and one fighter control officer were, on average, 42 Hz higher incombat than in non-combat phases during military flight exercises.

The mean increase in the average F0 results of individual pilotsvaried from 26 to 43%. The acoustic analyses were carried out byKeränenwith the samemeasurement methods and algorithms thatwere used in the present study. In our data, the relative difference inthe mean F0 values between cognitively loaded and less loadedflight phases was only around 10e12%. Regarding vocal intensity(sound pressure levels recorded inside oxygen masks), the meanvocal intensity measured by Keränen (2005) during in-flight non-combat phases was 129 dB, but only 124 dB in combat phases. Themean intensity of individual pilots’ speech varied from a decrease of1 dB to an increase of 3 dB. Unfortunately, a direct comparison of ourresultswith these results fromrealflights is not possible, because noestimation of cognitive loadwas carried out in the study by Keränen(2005), and because the phases of flight were classified somewhatdifferently in these two studies. However, we noticed that in ourdata collected in a simulator environment, average vocal intensitywas 15e20 dB lower compared with in-flight data from F/A-18Hornets. This probably reflects the effects of many physical, physi-ological, perceptual (especially noise) and psychological factors,including even higher cognitive load, which are present during realflights compared with what are found in a simulator environment.

4.6. Wider implications and suggestions for further studies

As to the practical implications of our results, information on theprosodic features of speech is usable in training and self-monitoring.Reliable detection of cognitive load-induced effects on voicemay beused during flight training, and in recognizing symptoms of declinein cognitive performance (Whitmore and Fisher, 1996), incapacita-tion in flight accidents (Saito et al.,1980; Australian Transport SafetyBureau, 2001), speaker state during high stress or workload or inother instances of high risk to flight safety (Clavel et al., 2008).Johannes et al. (2000) found F0 to be a potential variable in indi-cating progress in learning during training of work assignments inspace. In the future, neural networksmight proveuseful in detectingindividual distress or cognitive overload reflected in speech. Topromote flight safety, flight instructors and fighter control officerscould use data derived frompilot student’s or pilot’s speech to guidethem in their work assignments. Tools that employ neural network-based solutions could also be applicable, for example, in industry, indangerous assemblywork in remote locations, in forensic phonetics,in rescue operations and in driving and diving schools.

We found some individual variation in mean F0 change. In threeof our 13 participants virtually no change was seen in utterance-level mean F0 during the entire flight. Therefore, in the future itwould also be important to explore individual stress tolerance ordifferent reaction patterns associated with speech features. Forexample, Tolkmitt and Scherer (1986) found that personality type(low/high anxiety, anxiety denying) was associated with anincrease in the F0 floor during emotional stress. Additionally,Johannes et al. (2000), after having studied different clinical pop-ulations and astronauts/cosmonauts on a Mir space station,reported findings of voice pitch changing differently depending onthe typical physiological reaction pattern an individual generallyshows. The reaction patterns were associated with personalitytypes, a bit like Tolkmitt and Scherer (1986) proposed. Therefore, instudying the effects of cognitive load, collection of individualbaseline data on speech (and possibly data on personality type)seems to be an imperative for calibration of measurement systems.

We used two global measures of cognitive load: mean F0 andmean intensity values averaged over each short utterance. A morefine-grained analysis of F0 contour (like start and end points ofintonation contours), for example, might reveal interesting indi-vidual features of cognitive load. Further studies are needed tocover more fine-grained analyses, especially in natural speaking

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357356

situations. Johnstone and Scherer (2004), for example, suggestusing more complex variables that include dimensions of time,frequency and energy at the same time.

To conclude, we found bothmean F0 and vocal intensity changesto serve as rather good indicators of cognitive load inmilitary pilots.These variables could be used in monitoring applications in situa-tions where it is important to detect cognitive overload.

Acknowledgements

We kindly acknowledge Capt. Tommi Puro and Capt. SamuliNiemi (Satakunta Air Command, Finnish Air Force), and LanguageTraining Manager, Mr. Tapio Kakko and Capt. Taisto Puhakka (AirForce Command, Tikkakoski) for their help in data collection andpractical arrangements. This work was financed in part by theMinistry of Defense/Scientific Advisory Board of Defense, theFinnish Work Environment Fund and the Finnish Air Force.

References

Armentrout, J.J., Holland, D.A., O’Toole, K.J., Ercoline, W.R., 2006. Fatigue and relatedhuman factors in the near crash of a large military aircraft. Aviat. Space.Environ. Med. 77, 963e971.

Australian Transport Safety Bureau, 2001. Investigation Report 20000377. Pilot andpassenger incapacitation. Beech Super King Air 200 VH-SKC, WernadingaStation, Qld, 4 September 2000. Australian Transport Safety Bureau, Depart-ment of Transport and Regional Services, Canberra, Australia, pp. 1-22. Availablefrom: <http://www.atsb.gov.au/publications/investigation_reports/2000/AAIR/pdf/aair200003771_001.pdf>.

Batliner, A.R., Fischer, K., Huber, R., Spilker, J., 2003. How to find trouble incommunication. Speech Comm. 40, 117e143.

Berthold, A., Jameson, A., 1999. Interpreting symptoms of cognitive load in speechinput. In: Kay, K. (Ed.), Proceedings of the Seventh International Conference,UM99: User Modeling. Springer, Wienna, pp. 1e10. Available from: <http://www.dfki.de/wjameson/abs/BertholdJ99.html>.

Brenner, M., Doherty, E.T., Shipp, T., 1994. Speech measures indicating workloaddemand. Aviat. Space. Environ. Med. 65, 21e26.

Brockx, J., Nooteboom, S., 1982. Intonation and the perceptual separation ofsimultaneous voices. J. Phonet. 10, 23e36.

Casner, S.M., 2009. Perceived vs. measured effects of advanced cockpit systems onpilot workload and error: are pilot’s beliefs misaligned with reality? Appl.Ergon. 40, 448e456.

Clavel, C., Vasilescu, I., Devillers, L., Richard, G., Ehrette, T., 2008. Fear-type emotionrecognition for future audio-based surveillance systems. Speech Comm. 50,487e503.

Douglas-Cowie, E., Cowie, R., 2000. The vocal correlates of stress: a summary. In:Cowie, R., Douglas-Cowie, E., Schroeder, M. (Eds.), Proceedings of the ISCA ITRW:Speech and Emotion, 5e7 September 2000. Textflow, Belfast, pp. 219e224.Available from: <http://www.image.ntua.gr/oresteia/private/contributed_docs/reports/Voice-and-Stress.pdf>.

Elmenhorst, E.-M., Vejvoda, M., Maass, H., Wenzel, J., Plath, G., Schubert, E.,Basner, M., 2009. Pilot workload during approaches: comparison of simulatedstandard and noise-abatement profiles. Aviat. Space. Environ. Med. 80,346e370.

Endsley, M.R., 1993. A survey of situation awareness requirements in air-to-aircombat fighters. Int. J. Aviat. Psychol. 3, 157e168.

Endsley, M.R., 1994. Situation awareness: some reflections and comments. In:Gilson, R.D., Garland, D.J., Koonce, J.M. (Eds.), Situational Awareness in ComplexSystems. Embry-Riddle Aeronautical University Press, Daytona Beach, FL,pp. 315e317.

Endsley, M.R., 1995. Toward a theory of situation awareness in dynamic systems.Hum. Factors. 37, 32e64.

Endsley, M.R., 1999. Situation awareness in aviation systems. In: Garland, D.J.,Wise, J.A., Hopkin, V.D. (Eds.), Handbook of Aviation Human Factors. LawrenceErlbaum Associates, Mahwah, NJ, pp. 257e276.

Endsley, M.R., 2000. Theoretical underpinnings of situation awareness. In:Endsley, M.R., Garlands, D.J. (Eds.), Situation Awareness Analysis andMeasurement. Erlbaum, Mahwah, NJ, pp. 1e21.

Fernandez, R., Pickard, M., 2003. Modeling driver’s speech under stress. SpeechComm. 40, 145e159.

Flach, J.M., 1995. Situation awareness: proceed with caution. Hum. Factors. 37,149e157.

Grant, K.W., Walden, B.E., 1996. Spectral distribution of prosodic information.J. Speech. Hear. Res. 39, 228e238.

Griffin, G., Williams, C., 1987. The effects of different levels of task complexity onthree vocal measures. Aviat. Space. Environ. Med. 58, 1165e1170.

Hagmüller, M., Rank, E., Kubin, G., 2006. Evaluation of the human voice for indi-cations of workload induced stress in the aviation environment. EEC Note No.18/06, INO-1 AC STSS Project. EUROCONTROL Experimental Centre: Brétigny-

sur-Orge CEDEX, France. Available from: <http://www.eurocontrol.int/eec/public/standard_page/DOC_Report_2006_023.html>.

Hannula, M., Huttunen, K., Koskelo, J., Laitinen, T., Leino, T., 2008. Comparisonbetween artificial neural network and multilinear regression models in anevaluation of cognitive workload in a flight simulator. Comput. Biol. Med. 38,1163e1170.

Hannula, M., Koskelo, J., Huttunen, K., Sorri, M., Leino, T. 2007. Artificial neuralnetwork analysis of heart rate under cognitive load in a flight simulator. In:Deved�zic, V. (Ed.), Proceedings of the International Conference on ArtificialIntelligence and Applications as part of the 25th IASTED International Multi-Conference on Applied Informatics. February 12e14, 2007, Innsbruck, Austria.pp. 75e77.

Hansen, J.H.H., Swail, C., South, A.J., Moore, R.K., Steeneken, H., Cupples, E.J.,Anderson, T., Vloeberghs, C.R.A., Trancoso, I., Verlinde, P., 2000. The impact ofspeech under “stress” on military speech technology. RTO Technical ReportRTO-TR-10, AC/323(IST)TP/5 IST/TG-01, NATO Research & Technology Organi-zation, Neuilly-sur-Seine Cedex, France. Available from: <http://www.dtic.mil/cgibin/GetTRDoc?AD¼ADA377422&Location¼U2&doc¼GetTRDoc.pdf>.

Hart, S.G., Hauser, J.R., 1987. Inflight application of three pilot workload measure-ment techniques. Aviat. Space. Environ. Med. 58, 402e410.

Hirschberg, J., 2002. Communication and prosody: functional aspects of prosody.Speech Comm. 36, 31e43.

Hutchins, S.G., Morrison, G.J., Kelly, R.T., 1996. Principles for aiding complex militarydecision making. In: Proceedings of the Seventh International Command andControl Research and Technology Symposium., Monterey, pp. 1e18.

Jessen, M., Köster, O., Gfroerer, S., 2005. Influence of vocal effort on average andvariability of fundamental frequency. Speech Lang. Law 12, 174e213.

Johannes, B., Salnitski, V.P., Gugga, H.-C., Kirsch, K., 2000. Voice stress monitoring inspacedpossibilities and limits. Aviat. Space. Environ. Med. 71, A58eA65.

Johnstone, T., Scherer, K.R., 1999. The effects of emotions on voice quality. In:Proceedings of the 14th International Congress of Phonetic Sciences (ICPhS’99),San Francisco, pp. 2029e2032. Available from: <http://brainimaging.waisman.wisc.edu/wtjohnstone/0602.pdf>.

Johnstone, T., Scherer, K., 2004. Vocal communication of emotion. In: Lewis, M.,Haviland-Jones, J.-M. (Eds.), Handbook of Emotions. Guilford Press, New York,pp. 220e235.

Jones, C., Jonsson, I.-M., 2008. Using paralinguistic cues in speech to recogniseemotions in older car drivers. In: Affect and emotion in humanecomputerinteraction, in the series: Lecture Notes in Computer Science, vol. 4868.Springer, Berlin, pp. 229e240.

Junqua, J.-C., 1996. The influence of acoustics of speech production: a noise-inducedstress phenomenon known as the Lombard reflex. Speech Comm. 20, 13e22.

Keränen, H., 2005. Features of speech produced by Finnish military fighter pilots.Unpublished MA thesis in English Philology. Department of English, Universityof Oulu, Finland.

Keränen, H., Väyrynen, E., Pääkkönen, R., Leino, T., Kuronen, P., Toivanen, J.,Seppänen, T., 2004. Prosodic features of speech produced by military pilotsduring demanding tasks. In: Seppänen, T., Suomi, K., Toivanen, J. (Eds.), Fone-tiikan päivät 2004, Phonet Symp 2004. MediaTeam Oulu and suomen kielen,informaatiotutkimuksen ja logopedian laitos, Oulun yliopisto, Oulu, pp. 88e91.Available from: <http://www.mediateam.oulu.fi/publications/pdf/555.pdf>.

Kuronen, P., 2004. Military aviation noise. Noise-induced hearing impairment andhearing protection. Academic dissertation. Acta Universitatis Ouluensis, Medica,D 795. Oulu: University of Oulu (Finland). Available from: <http://herkules.oulu.fi/isbn9514274261/isbn9514274261.pdf>.

Lahtinen, T.M., Koskelo, J.P., Laitinen, T., Leino, T.K., 2007. Heart rate and perfor-mance during combat missions in a flight simulator. Aviat. Space. Environ. Med.78, 387e391.

Lee, Y.-H., Liu, B.-S., 2003. Inflight workload assessment: comparison of subjectiveand physiological measures. Aviat. Space. Environ. Med. 74, 1078e1084.

Lieberman, H.R., Niro, P., Tharion, W.J., Nindl, B.C., Castellani, J.W., Montain, S.J.,2006. Cognition during sustained operations: comparison of a laboratorysimulation to field studies. Aviat. Space. Environ. Med. 77, 929e935.

Lively, S., Pisoni, D., Van Summers, W., Bernacki, R., 1993. Effect of cognitiveworkload on speech production: acoustic analyses and perceptual conse-quences. J. Acoust. Soc. Am. 93, 2962e2973.

McCormack, H.M., de L. Horne, D.J., Sheater, S., 1988. Clinical applications of visualanalogue scales: a critical review. Psychol. Med. 18, 1007e1019.

Mendoza, E., Carballo, G., 1998. Acoustic analysis of induced vocal stress by meansof cognitive workload tasks. J. Voice. 12, 263e273.

Olsen, W.O., 1998. Average speech levels and spectra in various speaking/listeningconditions: a summary of the Pearson, Bennett, & Fidell (1977) report.Am. J. Audiol. 7, 21e25.

Prinzo, O.V., Britton, T.W., 1993. ATC/Pilot Voice Communications: a Survey of theLiterature. Final Report. Federal Aviation Administration, Oklahoma City, OK.

Protopapas, A., Lieberman, P., 1997. Fundamental frequency of phonation andperceived emotional stress. J. Acoust. Soc. Am. 101, 2267e2277.

Rothkrantz, L.J.M., Wiggers, P., van Wees, J.-W.A., van Vark, R.J., 2004. Voice stressanalysis. In: Text, Speech and Dialogue, in the series: Lecture Notes in ComputerScience, vol. 3206. Springer, Berlin, pp. 449e456.

Saito, I., Fujiwara, O., Utsuki, N., Mizumoto, C., Arimori, T., 1980. Hypoxia-inducedfatal aircraft accident revealed by voice analysis. Aviat. Space. Environ. Med.514, 402e406.

Scherer, K.R., Grandjean, D., Johstone, T., Klasmeyer, G., Bänziger, T., 2002. Acousticcorrelates of task load and stress. In: Proceedings of the 7th International

K. Huttunen et al. / Applied Ergonomics 42 (2011) 348e357 357

Conference of Spoken Language Processing (ICSLP2002), Denver, CO,pp. 2017e2020. Available from: <http://www.affective-sciences.org/system/files/2002_Scherer_Proc_ICSLP.pdf>.

Seppänen, T., Väyrynen, E., Toivanen, J., 2003. Prosody-based classification ofemotions in spoken Finnish. In: Eighth European Conference on Speech andCommunication Technology (Eurospeech 2003 e Interspeech 2003), Geneva,Switzerland, pp. 717e720.

Silberstein, D., Dietrich, R., 2003. Cockpit communication under high cognitiveworkload. In: Dietrich, D. (Ed.), Communication in High Risk Environments.Helmut Buske Verlag, Hamburg, pp. 9e56.

Skinner, M.J., Simpson, P.A., 2002. Workload issues in military tactical airlift.Int. J. Aviat. Psychol. 12, 79e93.

Stokes, A., Kite, K., 2003. Flight stress: stress, fatigue, and performance in aviation.Ashgate, Burlington.

Svensson, E., Angelborg-Thanderz, M., Sjöberg, L., Olsson, S., 1997. Informationcomplexitydmental workload and performance in combat aircraft. Ergonomics40, 362e380.

Svensson, A.I., Wilson, G.F., 2002. Psychological and psychophysiological models ofpilot performance for systems development and mission evaluation. Int. J. Aviat.Psychol. 12, 95e110.

Tolkmitt, F.J., Scherer, K.R., 1986. Effect of experimentally induced stress on vocalparameters. J. Exp. Psychol. Hum. Percept. Perform. 12, 302e313.

Traunmüller, H., Eriksson, A., 2000. Acoustic effects of variation in vocal effort bymen, women, and children. J. Acoust. Soc. Am. 107, 3438e3451.

Truong, K.P., van Leeuwen, D.A., Neerincx, M.A., 2007. Unobtrusive multimodalemotion detection in adaptive interfaces: speech and facial expressions. In:Schmorrow, D.D., Reeves, L.M. (Eds.), Augmented Cognition, HCII 2007, LNCS4565, pp. 354e363.

Wewers, M.E., Lowe, N.K., 1990. A critical review of visual analogue scales in themeasurement of clinical phenomena. Res. Nurs. Health. 13, 227e236.

Wickens, C.D., 2002. Situation awareness and workload in aviation. Curr. Dir.Psychol. Sci. 11, 128e133.

Whitmore, J., Fisher, S., 1996. Speech during sustained operations. Speech Comm.20, 55e70.

Wilson, G.F., 2002. An analysis of mental workload in pilots during flight usingmultiple psychophysiological measures. Int. J. Aviat. Psychol. 12, 3e18.

Wittels, P., Johannes, B., Enne, R., Kirsch, K., Gunga, H.C., 2002. Voice monitoring tomeasure emotional load during short-term stress. Eur. J. Appl. Physiol. 87,278e282.

Ylönen, H., Lyytinen, H., Leino, T., Leppäluoto, J., Kuronen, P., 1997. Heart rateresponses to real and simulated BA Hawk MK 51 flight. Aviat. Space. Environ.Med. 68, 601e605.

Zahorik, P., Kelly, J.W., 2007.Accuratevocal compensation for sound intensity losswithincreasingdistance innatural environments. J. Acoust. Soc.Am.122, EL143eEL150.