NCAR

Weather Technology in the Cockpit (WTIC) Transoceanic Human Over the Loop (HOTL) Demonstration

Final Test Report

Tenny Lindholm Cathy Kessinger Gary Blackburn Andy Gaydos

National Center for Atmospheric Research (NCAR)

10/1/2012

As long range and ultra-long range intercontinental flights become routine, weather information provided during preflight planning may not reflect reality when a flight needs hazardous weather information the most. The main motivator for this research is the need for hazardous weather information updates in data-sparse regions while the aircraft in en route. The demonstration described here is an initial step towards uplinking weather updates to actual aircraft so that the human factors and use case scenarios are better understood.

i

Table of Contents

Acknowledgments ......................................................................................................................................... v

Executive Summary ...................................................................................................................................... vi

1.0 INTRODUCTION ................................................................................................................................. 1

1.1 Background ................................................................................................................................... 1

1.2 Concept Scenarios ......................................................................................................................... 3

1.2.1 Selection of Flight Scenarios ................................................................................................. 3

1.2.2. Selection of Weather Scenarios ............................................................................................ 4

1.2.3. Flight Scenario for the Demonstration ................................................................................. 4

1.3. Support Tools ................................................................................................................................ 5

1.4. Purpose ......................................................................................................................................... 6

1.5. Reference Documents ................................................................................................................... 7

2. METHOD ................................................................................................................................................ 7

2.1. Participants ................................................................................................................................... 7

2.2. Research Personnel ....................................................................................................................... 8

2.2.1. FAA Aviation Weather Research, AJP-6850, WTIC Program ................................................. 8

1.2.2. FAA WJHTC ............................................................................................................................ 8

2.2.3. NCAR Research Applications Laboratory (NCAR-RAL) .......................................................... 8

2.3. Simulation Environment ............................................................................................................... 9

2.3.1. Research Facility .................................................................................................................... 9

2.3.2. Software, Weather Information............................................................................................ 9

2.3.3. Airspace ................................................................................................................................. 9

2.3.4. Traffic Environment (ATC) ................................................................................................... 10

2.3.5. Airline Operations Center (Dispatch) .................................................................................. 10

2.3.6. Research Cockpit Simulator (RCS) ....................................................................................... 10

2.3.7. Electronic Flight Bag ............................................................................................................ 13

2.4. Equipment ................................................................................................................................... 14

2.4.1. Communications ................................................................................................................. 14

2.4.2. Audio-Visual Recording System .......................................................................................... 14

2.4.3. Flight Deck Configuration .................................................................................................... 14

ii

2.5. Materials ..................................................................................................................................... 14

2.5.1. Informed Consent Statement ............................................................................................. 14

2.5.2. Pre-evaluation Questionnaire ............................................................................................. 14

2.5.3. Training Materials ............................................................................................................... 14

2.5.4. Preflight Information Bulletin (Weather, NOTAM Information) ......................................... 14

2.5.5. Departure, En Route Computer Flight Plan ........................................................................ 14

2.5.6. Post-evaluation Questionnaire ........................................................................................... 14

2.6. Experimental Design ................................................................................................................... 14

2.6.1. Independent Variables ........................................................................................................ 14

2.6.2. Simulation Measures .......................................................................................................... 16

2.7. Demonstration Procedures ......................................................................................................... 16

2.7.1. Overview ............................................................................................................................. 16

2.7.2. Training Session .................................................................................................................. 16

2.7.3. Evaluation Sessions ............................................................................................................. 17

Data Collection ........................................................................................................................................ 17

Testing/Demonstration Areas ................................................................................................................. 17

Testing/Demonstration Procedure Format ............................................................................................ 18

Flight One—Baseline ........................................................................................................................... 18

Flight Two--Enhanced ......................................................................................................................... 19

Post Mission De-brief .......................................................................................................................... 20

3. RESULTS AND DISCUSSION ...................................................................................................................... 20

3.1. Enhanced Weather Scenario Effectiveness Measures ................................................................ 21

3.1.1. Subjective Environmental Awareness ................................................................................. 21

3.1.2. Pilot Requests for Supplemental Information .................................................................... 21

3.1.3. Pilot Deviation Requests ..................................................................................................... 22

3.1.4. Subjective Safety Enhancement .......................................................................................... 24

3.2. Human Factors Measures ........................................................................................................... 25

3.2.1. Pilot Workload .................................................................................................................... 25

3.2.2. Controller Workload ........................................................................................................... 25

3.2.3. Airline Operations Center Workload................................................................................... 25

3.2.4. Situational Awareness Discussion ....................................................................................... 26

3.3. Observer Ratings ......................................................................................................................... 26

iii

3.4. Support Tool Effectiveness ......................................................................................................... 27

3.4.1. Preflight Information Bulletin ............................................................................................. 27

3.4.2. Airborne Weather Radar ..................................................................................................... 27

3.4.3. Electronic Flight Bag ............................................................................................................ 28

3.4.4. Enhanced Weather Information (EFB) ................................................................................ 28

3.4.5 RCS Out-the window (OTW) View (Figure 8) ...................................................................... 29

3.5. Discussion of Support Tool Improvements ................................................................................. 30

4. CONCLUSIONS ..................................................................................................................................... 31

References .................................................................................................................................................. 35

Acronyms .................................................................................................................................................... 36

Appendix A – Informed Consent Statement ............................................................................................... 37

Appendix B – Pre-evaluation Questionnaires ............................................................................................. 40

Appendix C – Computer Flight Plan and Scenario Script ............................................................................ 41

Computer Flight Plan .............................................................................................................................. 41

Scenario Script ........................................................................................................................................ 44

Appendix D – Post-evaluation Questionnaire and Observer Rating Form ................................................. 56

Appendix E – Pilot Training Handout .......................................................................................................... 58

Appendix F – Weather Scenarios for 2 July, 27 August, 2010 .................................................................... 63

iv

Table of Figures

Figure 1. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 1 June 2009 at 0115 UTC via an ASCII, line printer graphic (left) and a color-coded graphic (right) relative to the last known position of Air France Flight 447 (bottom center). The 30Kft contour is represented by a “/” and green shading; the 40Kft contour by a “C” and red shading The images are drawn relative to the expected flight route over the next 2 hours. ................................................................ 2 Figure 2. Flight scenario flow and information/communication interaction. Routine communications from dispatch and ATC will be simulated by the evaluator, consistent with the flight scenario. “Dotted” boxes represent input from the flight scenario or evaluator. .................................................................................. 3 Figure 3. The Spirit Airlines flight route is shown (red line) over the Cloud Top Height product (Kft, see color scale at right) on 2 July 2010 at 2045 UTC. Geographic boundaries are outlined with black lines. A convective SIGMET (red polygon) is shown over Florida............................................................................ 5 Figure 4. A photo of the ATC and AOC personnel and consoles used during the simulation..................... 10 Figure 5. A photo of the RCS during the simulation. .................................................................................. 11 Figure 6. A photo of the EFB is shown with the CTOP product displayed as an ASCII graphic (left) and as a color display (right). ................................................................................................................................ 12 Figure 7. A photo of the ND is shown at the bottom right. .......................................................................... 13 Figure 8. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 2 July 2010. Crew 1 (baseline) and Crew 2 (uplink) flight tracks for the entire route are shown. .... 22 Figure 9. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 27 Aug 2010. Crew 1 (uplink) and Crew 2 (baseline) flight tracks for the entire route are shown. .. 23 Figure 10. A photo of the First Officer’s ND and OTW view. ..................................................................... 30 Figure F-1. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 2 July 2010. Crew 1 (baseline) and Crew 2 (uplink) flight tracks for the entire route are shown…..61 Intermediate Figures F-2 – F-10 break the route down by waypoints and show the color uplink graphic (if available). Figure F-11. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 27 Aug 2010. Crew 1 (uplink) and Crew 2 (baseline) flight tracks for the entire route are shown...71 Figures F-12 – F-20 break the route down by waypoints and show the color uplink graphic (if available).

v

Acknowledgments

We would like to thank Mr. Ian Johnson, Federal Aviation Administration (FAA) Weather Technology in the Cockpit (WTIC) Program, for his helpful guidance with preparing for and conducting this research. Also, we would like to express our appreciation to Mr. Gary Pokodner and Mr. Eldridge Frazier, also from the FAA WTIC Program Office, for sponsoring this program. Ms. Mary Delemarre, Mr. Terence McLain, Mr. Tom Granich and Ms. Sam Fullerton (Engility Corp.), and their staff at the William J. Hughes Technical Center (WJHTC), plus the NextGen Integration and Evaluation Capability (NIEC) support staff, were absolutely crucial in coordinating the technical interfaces between NCAR and the WJHTC to enhance simulation realism. And so our thanks go to these professionals as well.

We cannot identify our pilot demonstration subjects by name. However, without exception, our pilot team members were outstanding in terms of being able to immerse themselves into the simulated scenarios and provide valuable critique and feedback during and after the demonstrations. We just could not ask for more. The pilots used during the two shakedown periods were most helpful as well, sharing their professional opinions and feedback leading to the most realistic Research Cockpit Simulator (RCS) performance possible. Our demonstration pilots were provided by HiTec Systems Inc. (Mr. Hal Olson).

The NCAR Research Applications Laboratory (RAL) team members include Ms. Cathy Kessinger and Mr. Tenny Lindholm, co-PIs; Mr. Gary Blackburn, software engineering lead; and Mr. Andy Gaydos, software engineer; plus other staff members at RAL. The spirit of cooperation between NCAR and the WJHTC was crucially important to working through the many technical challenges presented with merging the capabilities of both institutions and completing the research on schedule.

vi

Executive Summary

The recent Air France Flight 447 accident focused industry attention to the need for additional, aircraft-specific weather information in the cockpit, particularly for transoceanic flights. As long range and ultra-long range intercontinental flights become routine, weather information provided during preflight planning may not reflect reality when a flight needs hazardous weather information the most. The main motivator for this research is the need for hazardous weather information updates in data-sparse regions while the aircraft is en route. And because fleet-wide equipage for electronic flight bags (EFBs) and/or integrated flight displays will mostly lag technology capabilities, there is a need to portray the hazard information to the pilot using current avionics without modifying and certifying expensive modifications to primary flight displays and avionics. This research explores the concept of use, including potential training and human factors issues, of simple character graphic and color graphic depictions of rapidly updated weather information meant to supplement textual updates and airborne weather radar information.

The demonstration described here is an initial step towards uplinking weather updates to actual aircraft so that the human factors and use case scenarios are better understood. An actual air carrier trip from Ft. Lauderdale to Lima, Peru, was chosen for these simulation trials conducted in the WJHTC NIEC Research Cockpit Simulator (RCS) in Atlantic City, NJ. Actual weather scenarios were chosen from some 30 actual convective weather cases and simulated on a primary flight display (navigation display, or ND) for airborne weather radar information, and also simulated on an EFB in both a character graphic display format and a color graphic. The character graphic is meant to simulate an Aircraft Communications Addressing and Reporting System (ACARS) thermal printer display already installed on most Part 121 air carrier aircraft. The RCS was configured as an Airbus A-320 flight deck. Convective weather hazard information, based on satellite-based infrared cloud top data, was used for this initial study.

The results showed that the uplinked weather information was valuable in all aspects observed—crew situational awareness, workload reduction, more precise weather hazard avoidance, and crew decision-making. Further, the EFB character graphic was understandable and desired in place of no updates. The color graphic as presented on the EFB was preferred and very understandable. There were no safety issues identified as a result of the uplinked cloud-top height (CTOP) product. It was important, however, for flight crews to be trained on the use and interpretation of the information presented, including its limitations. Crew input and desires are documented in Chapter 3. A collateral benefit of this research was the development of an airborne radar display and simulation software that replicates actual weather specifically for the NIEC RCS. The airborne weather radar simulator is an important addition to the RCS in the NextGen research environment.

1

1.0 INTRODUCTION The Weather Technology in the Cockpit (WTIC) Program Transoceanic Human-Over-the-Loop (HOTL) Demonstration was conducted at the FAA Next Generation Integration and Evaluation Capability (NIEC) Research Cockpit Simulator (RCS) in the William J. Hughes Technical Center (WJHTC) located at Atlantic City, NJ, in July/August 2012. Four trials using two different current and qualified flight crews are described here, plus the background and motivation for this demonstration. Results are reported along with subjective assessments from the observers in Section 3. We also provide recommendations for further research using actual air carrier flights as part of our conclusions.

1.1 Background Prior to 2006, the FAA Aviation Weather Research Program (AWRP) sponsored the Oceanic Weather Product Development Team (OW PDT) which was charged to develop aviation weather products specifically designed to meet the needs of transoceanic aircraft. The OW PDT collaborated with United Airlines to successfully demonstrate the usefulness of an uplinked, satellite-based product that identified the 30kft and 40kft convective cloud top heights in a two-hour look-ahead display focused on the aircraft position and flight direction (like Figure 1). An ASCII character display was sent to the Boeing 777 aircraft onboard ACARS line printer when a significant amount of deep convection existed along the flight route. Similarly, the AWRP Turbulence PDT has demonstrated the uplink of a look-ahead turbulence severity product into the cockpit of selected CONUS United Airlines flights.

The recent Air France Flight 447 accident focused attention to the need for additional, aircraft-specific weather information in the cockpit, particularly for transoceanic flights. Figure 1 shows a reconstruction of the cloud top height product that could have been uplinked to the aircraft at the current position waypoint (shown as a red “X” in the bottom center of each panel), prior to its encounter with the mesoscale convective complex.

2

Figure 1. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 1 June 2009 at 0115 UTC via an ASCII, line printer graphic (left) and a color-coded graphic (right) relative to the last known position of Air France Flight 447 (bottom center). The 30Kft contour is represented by a “/” and green shading; the 40Kft contour by a “C” and red shading The images are drawn relative to the expected flight route over the next 2 hours.

The FAA, and airlines which frequently fly in oceanic and remote regions, are interested in exploring the usefulness of continuing the previous investigations into the economic and safety impacts of uplinked weather information. This capability also addresses NextGen goals toward a seamless transition from continental to oceanic flight while entering Oakland, New York, Anchorage, and Houston oceanic flight information regions (FIRs). However, prior to uplink of actual weather to en route aircraft, and to support development of a Concept of Use for uplinked oceanic weather, a focused ground demonstration of this capability using line airline pilots was deemed beneficial. The integration of “canned” weather information (that is, real weather cases but not live, real-time weather) into pilots’ decision processes in the ground simulator environment was planned using the FAA NIEC RCS. This ground evaluation focused the in-flight phase on the most relevant objectives identified during test planning.

3

1.2 Concept Scenarios

Figure 2. Flight scenario flow and information/communication interaction. Routine communications from dispatch and ATC will be simulated by the evaluator, consistent with the flight scenario. “Dotted” boxes represent input from the flight scenario or evaluator.

1.2.1 Selection of Flight Scenarios Flight routes were selected from a survey of actual airline trips that transit the inter-tropical convergence zone (ITCZ). This is a region approximately defined as ±30˚ latitude from the Equator where severe deep convection occurs frequently throughout the year. The flights that were considered generally originated from the East Coast of the U.S. with destinations in the Caribbean or South America. The planned flight scenario required a flight time of 4-5 hour duration. This allows a normal departure and climb to cruise altitude plus receipt of oceanic clearance. The en route portion of the flight was approximately three hours to allow flight encounters with two weather scenarios—that is, the presentation of two weather cases to the pilots for their evaluation and response. The flight scenario was terminated at the conclusion of the second flight encounter with weather, and execution of pilot decision(s) in response to the weather.

The demonstration flights required routes that transit the ITCZ because of the occurrence of frequent severe convection in that region. Severe convective activity in this region quite often is not accompanied by visual cues such as lightning, plus airborne weather radar returns may not fully indicate the presence of convective activity. Further, most air carrier flights through this

Preflt Dep/climb Oceanic clearance

En route

ATC, dispatch—response as needed

Weather updates—

pilot request, position report

PIB

• Voice • Graphical (EFB, text

graphic)

4

region are at night when avoidance of convective clouds cannot be aided through other visual cues. Other than this and the flight hour duration requirements, there were no other stipulations for the flight route. Computer flight plans for each flight scenario were developed and provided to the pilots. Corresponding preflight information bulletins (PIBs) were developed and provided, including typical weather briefing information that corresponds to the weather scenarios that were presented during the flight. Weather briefing materials were carefully prepared to include the normal level of uncertainty that would be expected from a forecast that is prepared 6+ hours prior to the actual flight encounter.

1.2.2. Selection of Weather Scenarios Weather scenarios, or cases, were selected from actual convective weather events that were typically seen within the ITCZ. They were carefully chosen so that a pilot decision to avoid the affected area would be expected. This means the convective weather event should have sufficient horizontal coverage and vertical extent that the planned flight profile would be significantly impacted by the weather presented in the scenario.

Thirty weather cases were initially identified from data sets archived at NCAR-RAL. These were analyzed for potential in meeting the above criteria, and down selected to the three cases used for the demonstration. The scenario data sets that were chosen were integrated into the aircraft airborne radar simulation and cloud top height (CTOP) character and color graphic products. They were:

• 16 August 2010, used during the pilot training syllabus • 2 July 2010, scenario #1 for the demonstration runs • 27 August 2010, scenario #2 for the demonstration runs

A CTOP product was generated for each weather scenario. The CTOP provided in the update was a diagnosis of convection that was valid at the time of the pilot request (in terms of simulation time) or when a position report was received from the pilot. The position report was used to generate the CTOP product as described here and uplink of information in the absence of a specific request from the pilot. That is, a product “push” concept was used by default.

1.2.3. Flight Scenario for the Demonstration The following demonstration factors were identified and incorporated into the selection of the flight scenario:

1. First, due to the limited test cases allowed for this effort, statistics on crew responses were not sufficient to draw complete conclusions on the objectives. Rather, observations on crew responses, weather avoidance criteria, safety, and procedures were collected and are presented in this demonstration final report.

2. Flight duration of 4-5 hours was required. 3. Flight route must transit the ITCZ. 4. Location of the flight route, other than demonstration factors 2. and 3. from above, was

not an important demonstration requirement. Likewise, type of aircraft, airline

5

Operational Specification (OPSPEC), and city-pair were not relevant to the demonstration.

5. A simulated night flight was desired, as this would remove the need to correlate visual cues (other than lightning) with on-board weather displays. The worst case scenario for weather detection and avoidance is a night, overwater, moonless flight condition,

The route selected for the demonstration was a Spirit Airlines A-320 Part 121 operation:

Spirit Airlines Ft. Lauderdale Lima, Peru 5:25PM 10:07PM 4+42 hrs

Figure 3 shows a plan view of the flight route over the CTOP product on 2 July 2010 at 2045 UTC.

Figure 3. The Spirit Airlines flight route is shown (red line) over the Cloud Top Height product (Kft, see color scale at right) on 2 July 2010 at 2045 UTC. Geographic boundaries are outlined with black lines. A convective SIGMET (red polygon) is shown over Florida.

1.3. Support Tools The following elements were essential to completing the oceanic weather demonstration in the NIEC RCS:

6

1. Airline pilots who were experienced in the aircraft type and oceanic/remote flight operations were provided as evaluation subjects.

2. The NIEC Laboratory RCS capabilities that were required and provided are: a. The NIEC RCS was configured as an Airbus A-320/330 aircraft b. The aircraft flight management system (FMS) was partially functional. Because

of a protective plexiglass shield over much of the center console, parallax error and touch sensitivity made data entry difficult. The FMS was pre-loaded with the flight plan, and did update as waypoints were passed. Fuel planning pages were working, but changes to FMS pages were difficult and really not relevant to the demonstration. The ACARS was operational from both the FMS and dispatch.

c. Panel switches/controls were touch-activated on flat panel displays d. The simulator was a Class 4 simulator, allowing for realistic flight scenarios from

gate pushback through en route operations e. The simulator was not FANS-1 capable but allows for HF ATC comm./position

reporting f. Aircraft position was known (lat/long) at all times to support tailoring of

satellite-based weather hazard information g. ATC and airline operations center (AOC) communications was simulated as

needed in response to pilot requests h. The simulator was equipped with a simulated EFB i. The NIEC RCS allowed ingest of “canned” weather data, and display on flight

displays and EFB j. The NIEC RCS could accommodate any global flight scenario

3. Actual weather scenarios were identified and processed so that they could be ingested into the NIEC RCS and viewed on the flight deck displays.

4. Trip-specific PIBs and flight folders (with weather planning information) were provided to the evaluation subjects during preflight training.

5. Trip scenarios that address the demonstration objectives were ingested into the NIEC RCS to replicate as much as possible an actual airline flight.

6. Scenario-specific weather information to respond to pilot requests were prepared and provided to the pilot on request.

1.4. Purpose The purposes of the Transoceanic HOTL Demonstration using the FAA NIEC RCS include:

• Evaluate the risk of in-flight evaluations of updated weather information in oceanic/remote regions.

• Increase the understanding of impacts to pilot, dispatch, and air traffic management (ATM) decision making in a collaborative environment when updated oceanic weather information is provided to the flight deck.

• Identify demonstration objectives that are best accomplished with an expanded demonstration of uplinked hazardous weather information to transoceanic airline flights.

Demonstration objectives are to:

7

• Train flight crews on the capabilities and limitations of planned uplink weather and representations presented on the flight deck. Identify content needed in the flight demonstration Flight Crew Bulletin (FCB) and crew training. Uplinked weather information in oceanic/remote areas may be limited in terms of information content, spatial and temporal resolution when compared to that provided in less data-sparse environments in continental regions.

• Identify those decisions pilots make in the current environment without weather updates, and propose decisions that can be facilitated with frequent weather hazard updates while en route in oceanic/remote regions. Specifically

o How does updated weather information affect timing of deviation request from the pilot?

o How does the updated information enhance operational safety? That is, does the availability of updated weather hazard information (in addition to that provided by the airborne radar) decrease the flight’s exposure (as measured by radar range) to hazardous weather during an encounter?

o Does a timely update result in a reduction of flight time and/or fuel burn? o Does the passive uplink of weather updates reduce pilot communications with

dispatch and with air traffic control? • Obtain initial flight crew feedback on weather hazard needs and display presentation

concepts. Specifically, o Is the ACARS text graphic adequate for conveying basic hazard information? o Does the increased potential for information transfer offered by a graphic

electronic flight bag (EFB) display provide additional efficiency and safety benefit?

• Identify situations where collaborative decisions between air traffic controllers, dispatch, and pilots using common, updated oceanic weather hazard information can benefit operations in oceanic/remote regions.

• Build upon demonstration objectives that can be addressed from the uplink of updated weather hazard information (specifically, turbulence and convection) to operational airline flights in oceanic/remote regions.

1.5. Reference Documents There are no specific reference documents for this demonstration. All procedures used during the flight scenarios were in accordance with establish flight information publications, International Civil Aviation Organization (ICAO) procedures (ICAO Annex 3 for weather information), and airline operational specifications and procedures.

2. METHOD

2.1. Participants The WJHTC NIEC Laboratory provided operational airline pilots who were current and qualified to accomplish the planned flight scenarios in the A-320 aircraft. Actual qualification and experience levels are reported in Section 3. Two flight crews were provided for the one-week demonstration. Each crew flew two instances of the flight scenario with different weather

8

scenarios for each flight. Pre- and post-flight briefings included necessary training (please see Appendix B for the preflight questionnaire and Appendix E for the training handout) and an in-depth discussion that addressed the objectives. Airline flight crews provided by the WJHTC NIEC Laboratory also participated in the scheduled NIEC RCS shakedown periods and simulator pre-flight training sessions.

The FAA provided qualified personnel to perform both the ATC and airline operations center (AOC) or dispatch functions.

2.2. Research Personnel

2.2.1. FAA Aviation Weather Research, AJP-6850, WTIC Program The FAA WTIC Program sponsored the NIEC Oceanic HOTL Demonstration and provided programmatic and program management resources for this effort, potentially leading to an in-flight demonstration on operational airline flights. The HOTL phase of the demonstration is desired to enhance understanding of the crew interface and use of oceanic weather updates. All changes to the baseline plan required approval by Weather Research Branch Manager ANG-C61 or designee.

2.2.2. FAA WJHTC The WJHTC provided the technical support for the Oceanic HOTL Demonstration through the NIEC Laboratory and RCS, in response to the requirements set forth by the FAA WTIC Program Office. Specifically, the NIEC:

• Integrated flight scenarios for the demonstration. These were actual airline flights through the inter-tropical convergence zone with duration of 4-5 hours.

• Allocated NIEC RCS time (schedule in Section 6) for four simulator missions, each four hours in duration. Two additional four hour simulator periods were allocated for pre-mission training and familiarization.

• Provided for the collection of data as defined by NCAR and the demonstration team (described in Section 3).

• Scheduled airline flight crews to fly the simulator missions. One flight crew was required to “shake-down” the simulator and validate the scenarios prior to the demonstration. Two different flight crews were used for the demonstration—each demonstration crew flew two simulator missions.

• Provided technical assistance for the preparation of the demonstration plan, and operational expertise from staff pilots to ensure scenarios are realistic and relevant.

• Assisted with formatting and ingest of flight and weather scenarios into the NIEC RCS. • Conducted flight crew training on the NIEC RCS.

2.2.3. NCAR Research Applications Laboratory (NCAR-RAL) NCAR-RAL was responsible for planning and execution of the HOTL demonstration of oceanic weather hazard information uplink/display to pilots. This included

1. Completion of a test plan for the demonstration.

9

2. Preparation of flight crew training materials and completion of crew training on weather information and displays. Training materials were made available to FAA Flight Standards for review (Appendix E).

3. Preparation of weather scenarios for the demonstration. This included the weather hazard update information formatted as needed for ingest into the simulator. RAL also prepared textual and verbal weather information consistent with the weather scenarios and available to support pilot requests while en route.

4. Preparation of pre-flight weather and PIB for NIEC simulated scenarios. Scenario materials were made available to FAA Flight Standards for review prior to shakedown and evaluation missions.

5. Development of display concepts for both text graphics and color graphics. 6. Software engineering interface with NIEC to determine how and plan for integration of

weather into the NIEC RCS for EFB and character graphic displays. 7. Collecting and analyzing data (as described in Section 3) and preparation of a final

report. 8. Future planning for the in-flight phase of the demonstration (based on a decision by the

Aviation Weather Group Management Team).

2.3. Simulation Environment

2.3.1. Research Facility The NCAR Research Applications Laboratory facilities and computer resources were used to generate weather depictions for the navigation display (ND) and EFB.

The WJHTC NIEC and RCS were used for the actual demonstration flights.

Detailed descriptions of facilities are in Section 1.3.

2.3.2. Software, Weather Information EFB weather depictions were created (similar to those shown in Figure 1) in response to position reports at compulsory reporting points (CRPs). Software was developed, maintained and operated remotely at NCAR. CTOP products were created for the current position, plus the next two CRPs. The products were then integrated with the planned flight profile for the trip scenario for display on the EFB. The ND airborne weather display was also fed from RAL software. These software scripts replicated what would be displayed on the ND given the weather scenarios for the trip, including the potential for radar attenuation caused by scattering of radar energy reflected from the storms. Attenuation may cause shadowing that hides reflectivity from storms behind the leading edge of a storm area.

2.3.3. Airspace Airspace chosen for the HOTL demonstration simulated an oceanic, remote flight outside of surveillance and weather radar coverage (even though, in reality, the route flown would be under surveillance for most of the flight).

10

2.3.4. Traffic Environment (ATC) For the HOTL demonstration, ATC interaction was provided by subject matter expert(s) (SME) so that the pilot could ask for assistance with reroutes or request weather information (Figure 4). Position reporting was in accordance with ICAO procedures for CRPs along the route of flight. Air traffic control and management procedures were not a factor, as the simulated flight profile was not impacted by other air traffic. Flight profile requests were approved as needed.

Figure 4. A photo of the ATC and AOC personnel and consoles used during the simulation. 2.3.5. Airline Operations Center (Dispatch)

Airline operations center (AOC), or dispatch, interaction was provided by SME(s) (Figure 4). Standard ICAO weather information was provided to the AOC so that the pilot could request updated textual/verbal weather as desired.

2.3.6. Research Cockpit Simulator (RCS) The NIEC RCS cockpit configuration took advantage of existing capabilities as much as possible so that reconfiguration and revalidation were not required. The subject flight crews were familiar with the flight procedures and cockpit functionality of the demonstration configuration to eliminate any training requirement beyond basic familiarity with the NIEC RCS.

11

Figure 5. A photo of the RCS during the simulation.

Demonstration objectives required the following basic configuration and functionality present on a typical Airbus 320/330 flight deck:

1. Actual EFBs (Astronautics NEXIS) were located on the outboard side panel for each pilot. The EFB was used for the presentation of both ASCII and graphic weather update information. ACARS text messaging between the pilot and AOC was accomplished using the Flight Management System (FMS) Multi-function Control and Display Unit (MCDU) on the center console, with voice communications as a backup (phone patch via the air navigation service provider (ANSP)). The EFB also displayed passive weather character and color graphics as they were provided in accordance with the flight scenario. Figure 6 shows a photo of the EFB with a cloud top height graphic.

12

Figure 6. A photo of the EFB is shown with the CTOP product displayed as an ASCII graphic (left) and as a color display (right).

2. The airborne weather radar display was operational and functional so that the pilots are able to corroborate information between weather depictions on the EFB and radar display (consistency). The weather radar depiction was on the ND consistent with the A-320 configuration. (Figure 7)

13

Figure 7. A photo of the ND is shown at the bottom right.

3. Aircraft position on the navigation display was available at all times to the pilots, and was consistent with the airborne weather radar depiction that was generated by the weather scenario. Updates of the RCS ND weather depiction did not precisely model the actual sweep of the airborne radar. That is, ND updates were on the order of every minute vs. continuous. Also, the tilt feature was not available on the ND.

4. The EFB showed aircraft position at the time of creation of the character or color graphic for the weather update.

5. The EFB showed both character graphics and color graphics to facilitate comparative evaluations of the two capabilities. That is, the pilot was able to select either display as needed to support the demonstration objectives. Paper backups of the EFB displays were available to enable detailed discussion of the weather scenarios during and after the flight.

6. Paper aeronautical charts and corresponding computer flight plan (CFP) were provided to the pilots as needed to complete the flight scenario. Pilot navigation workload was not adversely affected by en route navigation procedures that were unfamiliar to the evaluation pilots.

ACARS was the primary method to communicate with company dispatch for the baseline

and enhanced test flights. A functioning ACARS added realism to the simulation and replicated the workload the average line pilot would experience in managing these weather events (for this routing.)

2.3.7. Electronic Flight Bag See previous Section 2.3.6.

14

2.4. Equipment

2.4.1. Communications The pilots had voice communications available to AOC and ATC, plus the FMS MCDU had texting capability to the AOC. VHF voice communications were used for the flight in accordance with the ICAO flight procedures for the Latin and South America regions.

2.4.2. Audio-Visual Recording System An audio-visual recording capability was available and used during the demonstration.

2.4.3. Flight Deck Configuration See Section 2.3.6.

2.5. Materials

2.5.1. Informed Consent Statement Contained in Appendix A.

2.5.2. Pre-evaluation Questionnaire Contained in Appendix B.

2.5.3. Training Materials Contained in Appendix E.

2.5.4. Preflight Information Bulletin (Weather, NOTAM Information) The PIB for the scenario was provided to the pilots during preflight preparations. This package included normal ICAO-prescribed weather information for the flight.

2.5.5. Departure, En Route Computer Flight Plan The most current CFP available from Spirit Airlines was used for the scenario script and route of flight. Both are contained in Appendix C.

2.5.6. Post-evaluation Questionnaire The post-evaluation questionnaire and observer rating form are contained in Appendix D.

2.6. Experimental Design

2.6.1. Independent Variables To fully understand the enhanced operational capabilities provided by updated weather via data link and resulting HOTL interaction, we must first document current pilot procedures and their effect on pilot decision-making. NextGen envisions an optimal mix of human-over-the-loop (HOTL) decision-making and automation, and a seamless transition from continental to oceanic flight. Current procedures are largely human-in-the-loop (HITL) during en route flight segments over oceanic/remote areas. So our understanding of the contrasts between HITL and HOTL pilot involvement in the decision-making process is a valuable outcome of this demonstration.

15

Previous demonstrations of the uplinked weather update capability have used a character graphic printout on the existing ACARS thermal printer on the flight deck to convey weather hazard information. The character graphic (Figure 1) was devised to quickly and cheaply demonstrate the value of weather information updates to more effective decision-making. Past demonstrations have uplinked both convection and turbulence updates. The EFB concept with varying levels of capability and complexity provides a pilot interface that allows color graphic depictions of weather hazard information, plus more precise own-ship position. Another important outcome of this demonstration is to obtain pilot feedback on the relative merits of the character graphic and color graphic depictions.

Therefore, there are two dimensions to the flight scenario “experimental design” that this effort will demonstrate:

1. Baseline (no updates) verses weather updates provided, and 2. Character graphic verses color graphic.

The design devoted one entire flight to the baseline case, and one flight to the enhanced weather update capability for each flight crew. The same flight route was used for each demonstration to eliminate any retraining based on the route procedures between demonstration flights. Further, two different flight crews experienced both flight scenarios so that demonstration observers can identify any repeatability in procedures and learn further about possible unique pilot decision-making techniques in both the baseline and enhanced scenarios.

As discussed above, weather scenarios were chosen from actual weather cases that would be severe enough to presume a pilot response (re-route). In the baseline case, pilots had the airborne weather radar available to provide the initial stimulus for a possible re-route. These radars typically provide (realistically) a 150-320 nm look-ahead capability. In the enhanced case, pilots were provided an “uplinked” weather update 400-550 nm prior to the event. The airborne weather radar was available as well, providing its normal look-ahead capability. In both cases, the pilots had a pre-flight PIB available and were able to coordinate with dispatch and ATC as needed. The flight scenarios (3+ hours en route) allowed exposure to two weather cases on each flight. The selected weather scenarios were two separate weather cases. Each crew was presented with both weather cases according to the following design:

Flight 1 Crew 1 Baseline Weather Scenario 1 Flight 2 Crew 1 Enhanced Weather Scenario 2 Flight 1 Crew 2 Baseline Weather Scenario 2 Flight 2 Crew 2 Enhanced Weather Scenario 1

Therefore, for each flight crew, the demonstration consisted of two 4-hour flights:

16

1. Flight 1: Ft Lauderdale to Lima Peru, current procedures (no updates other than normal ATC and dispatch communications). Normal takeoff, climb. En route weather case(s) presented during cruise. Baseline case.

2. Flight 2: Ft. Lauderdale to Lima Peru, enhanced procedures (weather updates provided, with normal ATC and dispatch communications). Normal takeoff, climb. Weather updates provided every 30 minutes to 1 hour. En route weather case(s) presented during cruise. Enhanced case.

The scenario was repeated for the second demonstration flight crew.

2.6.2. Simulation Measures Observations, subjective assessments by participants and observers, and objective measurements are described below in the context of the flight scenarios. The post-demonstration questionnaire and observer form were used to document these assessments.

2.7. Demonstration Procedures

2.7.1. Overview The simulated environment resembled a line-oriented flight training (LOFT) mission on an oceanic flight.

1. Pre-demonstration planning included representatives from FAA WTIC Human Factors team, the NIEC team, and NCAR. This period included integration of real weather from archived weather data collected by NCAR-RAL.

2. Oceanic air traffic control subject matter expert (SME) and FAA Flight Standards SME assisted in verifying the realism of the flight scenarios during RCS shake-down periods. The SMEs looked at the route from an ATC perspective; ensuring ground stations have appropriate scripts to respond to the flight as it progressed through Flight Information Region (FIR) transitions and as the pilots transmitted position reports and requests.

3. Actual airline flight crews were used with the appropriate experience. 4. The demonstration consisted of two simulator sessions for each flight crew (four sessions

total). The first flight (for each crew) was a baseline case; the second added enhanced weather information that was presented on the EFB. Sessions were four hours long. Demonstration pilots received training on the unique aspects of the NIEC RCS and the capabilities/limitations of the weather information that was presented.

5. For each flight crew, a scenario was presented with weather information as currently available, and then with updated weather hazard information on a graphical display (ACARS character graphic and color graphic on the EFB). Decision making behavior and feedback from pilots were recorded and used to establish a concept of use baseline for Operational Demonstrations on actual oceanic flights.

2.7.2. Training Session Two flight crews participated in the NIEC RCS demonstration, each crew completing one day of training/orientation and two days of formal evaluation flights.

Pre-mission training and briefing (notionally, four hours) accomplished the following:

1. Familiarize the pilots with the NIEC RCS’s capabilities and limitations.

17

2. Review airborne radar use, limitations in the baseline case scenario (for the benefit of both the demonstration pilots and evaluators).

3. Review baseline sources of weather information while on route, and limitations. 4. Train the flight crews on the information content and limitations of enhanced case

weather updates, to include demonstration procedures for contract and on-demand weather updates. Contract updates were provided in response to position reports given during flight outside of radar coverage. On-demand updates were provided in response to a pilot request.

5. Review demonstration procedures and flight scenarios.

Pre-mission sessions were conducted in the Developmental RCS cockpit simulator at WJHTC (Bldg. 201), so that pilots can become more familiar with the flight deck configuration and functionality.

2.7.3. Evaluation Sessions

Data Collection

In general, it was deemed useful to obtain pilot input on what other weather hazard graphic depictions, plus functionality, should be added to the EFB for further evaluations (for example, vertical cross-section, turbulence, jet stream). This feedback from line pilots is important and was solicited both during and after each simulator session. Other data collected and documented for the final report include:

1. Observations and questions listed below (NCAR). 2. Pilot voice and/or data communications (ACARS) with dispatch and ATC (number of

pilot initiated requests for information, NCAR). 3. Recording of actual deviations around weather that were initiated by the pilot without

ATC clearance. Often, in oceanic regions, pilots will deviate within course tolerance (±5-10 nm) for weather avoidance without coordinating with ATC (number of deviations, magnitude of deviations, NIEC).

4. Cockpit video (en route phase, NIEC). 5. Instances of and techniques used for cockpit resource management and crew coordination

(NCAR observers). 6. Pilot background, experience (NCAR, questionnaire).

A notional one-hour de-briefing period followed each flight demonstration. Any further discussion or pilot feedback was documented for the final report.

Testing/Demonstration Areas

Results from the simulated LOFT provided a “concept of use baseline,” providing the FAA and NCAR with insights into:

1. Current oceanic flight operations with pre-flight pilot information briefings (PIBs) and available weather updates;

18

2. Limitations to decision support for pilots, dispatchers, and air traffic managers posed by the current operational environment;

3. Weather hazards related to flight safety issues;

4. Potential changes to the operational procedures when in-flight updates of weather are available on a contract, demand, or broadcast service;

5. Potential impacts to ATM and safety.

6. Refined objectives for a future operational oceanic flight demonstration. Testing/Demonstration Procedure Format

The HOTL/LOFT simulator missions approximate as much as possible an actual oceanic airline flight. Repositioning of the simulator was not done so that pilot task flow remained realistic. Since the demonstration objectives were addressed mostly through observations by the evaluators, the interruption of flight progress was allowed so that the pilots could share their opinions of the value of weather information, display presentation, or reasons for selecting a particular course of action. Detailed discussions of aspects of the demonstration were done post-mission as much as possible.

NCAR and WTIC Program researchers acted as observers for the demonstration. NIEC pilot experts also participated.

Flight One—Baseline Each evaluation mission was preceded by a 1-hour pre-mission briefing. This time was used to review evaluation procedures and pilot expectations from weather information, ATC, and airline dispatch. During the flight scenario, normal flight procedures were used and observed through climb-out and level-off. The flight scenario included the following procedures and normal ATC/dispatch communications:

1. Checklist through before take-off checklist. 2. Take-off clearance, lineup checklist, normal take-off. 3. After take-off/climb procedures and checklist. Once the aircraft was established en route, the notional sequence of events for the baseline

flights was (Figure 2):

1. The first weather scenario was presented on the airborne weather radar, approximately 150-320nm prior to the leading edge of the event (consistent with the radar look-ahead capability).

2. Pilot response—request for weather updates via text or verbal, pilot reference to PIB information, pilot request for reroute or altitude change.

3. Presentation of second weather scenario. 4. Pilot response. 5. Resumption of en route procedures until clear of weather.

19

• Observations • Reference to PIB information • Pilot requests for information from dispatch and ATC—what and when? • Pilot requests for reroute or deviation. When/how early does the stimulus result in

a perceived need to deviate? How many deviations are required? • What was the flight’s closest approach to the weather hazard? • What was the cost in terms of flight time (fuel) resulting from the deviation

strategy? • What was the workload impact caused by the weather scenario? One measure of

this might be the number of attempts to request information (either via HF voice or ACARS data communications) from dispatch and/or ATC, or reference to on-board PIB information.

• Questions • Subjective assessment of workload impact. • Subjective assessment of safety impact. • What other information would the pilot seek in the baseline case? • What more information would have helped the decision-making process in the

baseline case? Flight Two--Enhanced Each evaluation mission was preceded by a 1-hour pre-mission briefing. This time was used to review evaluation procedures and pilot expectations from weather information, ATC, and airline dispatch. During the flight scenario, normal flight procedures were used and observed through climb-out and level-off. The flight scenario included the following procedures and normal ATC/dispatch communications:

1. Checklist through before take-off checklist. 2. Take-off clearance, lineup checklist, normal take-off. 3. After take-off/climb procedures and checklist.

Once the aircraft is established en route, the notional sequence of events for the enhanced

flights was (Figure 1):

1. The first weather scenario was presented in the form of a forced uplink, a minimum of 400nm from the leading edge of the weather event. The uplink was in response to a pilot level-off position report and estimated time of arrival for the next waypoint. In a FANS environment, this report would be automatic. The weather scenario was presented in both character graphic and color graphic formats on the EFB.

2. Pilot response—request for weather updates via text or verbal, pilot reference to PIB information, pilot request for reroute or altitude change.

3. Corroborated weather information was presented on the airborne weather radar display. 4. Additional pilot response to corroborated evidence of the weather hazard. 5. Presentation of second weather scenario. 6. Pilot response. 7. Resumption of en route procedures until clear of weather.

20

• Observations

• Reference to PIB information • Pilot requests for information from dispatch and ATC—what and when? • Pilot requests for reroute or deviation. When/how early does the stimulus result in

a perceived need to deviate? How many deviations are required? • What was the flight’s closest approach to the weather hazard? • What was the cost in terms of flight time (fuel) resulting from the deviation

strategy? • What was the workload impact caused by the weather scenario? One measure of

this might be the number of attempts to request information from dispatch and/or ATC, or reference to on-board PIB information.

• Questions • Subjective, comparative assessment of workload impact relative to no update. • Subjective, comparative assessment of safety impact relative to no update. • What other information would the pilot seek beyond the weather update? • What more information on the EFB would have helped the decision-making

process? • Subjective contrast (understanding, information transfer, usability) between the

character and color graphic. • Would more altitude contours enhance decision-making? • What additional pilot interaction, display functionality would enhance

information transfer? Post Mission De-brief At the conclusion of each simulator session, the NCAR observers conducted a post-mission debriefing. The debriefing time (nominally, one hour) was spent clarifying pilot questions and NCAR observations during the session. The pilots were given the opportunity to fill out a questionnaire that was focused on the demonstration objectives and the RCS’ utility as a demonstration or evaluation tool for HOTL methodology development.

3. RESULTS AND DISCUSSION The following observations and feedback came from pilot decisions during the flight scenarios, pilot actions, pre- and post-demonstration questionnaire comments, and verbal comments directly from the pilots. The four pilots were provided by the WJHTC (HiTec Systems, Inc.) and were active A-320 pilots with oceanic experience. Total flight experience of these pilots ranged from 17,000 to 21,000 flight hours. All of them also had experience with Caribbean and South American flight operations.

21

3.1. Enhanced Weather Scenario Effectiveness Measures

3.1.1. Subjective Environmental Awareness Pilots were asked to compare their overall situational awareness between current oceanic operations and the enhance weather update case, and all rated the enhanced case “much more effective.” Some anecdotal evidence supporting this subjective rating included:

• One pilot stated the ND radar display “painted us into a corner,” and having been exposed to the CTOP graphics during training commented that he “missed not having this information” during the baseline scenario.

• “The best value of this is the ability to look behind a storm area” to ascertain the potential for attenuation. This pilot prefaced most of his decisions with an assessment of the attenuation potential during the enhanced flight.

• “In the real world, this radar is only good out to 160nm.” The CTOP benefit is to supplement the airborne radar. This pilot further stated the value of the CTOP information is “greatest when tactical maneuvering using the radar, and with CTOP in-hand.”

• Pilots, in several cases, decided on deviating (baseline scenarios) not knowing what was beyond 160nm. The result was a track that was greater than 100nm off course. One deviation resulted in a 150nm off-course situation. It happens that 160nm is the observed break-point between tactical avoidance and strategic deviation.

One pilot stated that he always flies with what his airline calls an “orientation chart,” which is simply a strip map that covers the route of flight plus ETOP and terrain “escape route” information. He suggests that weather hazards be integrated into the orientation chart to give a complete picture of the flight route from departure to destination.

3.1.2. Pilot Requests for Supplemental Information Pilots universally thought that requests for supplemental information from dispatch might increase given the updated weather information. This might be expected since the pilots are becoming nearly as aware of the weather situation as the dispatchers, and so the uninformed request is replaced by an aware, informed discussion. One pilot verbally commented on his observation that pilots like confirmation of sources of information which in turn reduces uncertainty. Dispatch communication did in fact increase during the enhanced flight scenarios.

ATC, in general, was not thought to be a primary source of updated weather information except for routine ride reports. Requests for deviations from ATC were usually accompanied by a request for confirmation of the need to deviate, not necessarily what deviation is best. That request went to dispatch.

Other information requested from dispatch included whether a re-dispatch was needed to change or eliminate alternates given an arrival over destination at least 2000lb light on fuel (this occurred each time the flight exceeded 100nm in deviation from planned course).

22

3.1.3. Pilot Deviation Requests When asked to summarize the number of deviation requests, the answers ranged from “too many to count” to over 50. The pilots all felt that the enhanced CTOP information allowed for more “strategic” deviations and hence fewer than when tactically avoiding the storms using airborne radar set at 40-80nm range. Heading changes during the tactical maneuvering phase were just done as needed to “thread the needle.” Larger, more strategic heading changes were done in coordination with ATC. One pilot stated that even though the more strategic turns off-course might have resulted in larger diversions from course, there was a trade-off with enhanced safety by avoiding the storm areas by a larger distance.

Figure 8. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 2 July 2010. Crew 1 (baseline) and Crew 2 (uplink) flight tracks for the entire route are shown.

23

Figure 9. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 27 Aug 2010. Crew 1 (uplink) and Crew 2 (baseline) flight tracks for the entire route are shown.

Figure 8 and Figure 9 illustrate pilot responses from post-analysis of flight track data and weather scenarios that were presented. These flight track plots are broken down by waypoints for the four weather scenarios in Appendix F, The color graphic presentation (when available) that was derived from the satellite CTOP for each flight segment is included.

Figure 8 is the 2 July 2010 weather case. Flight tracks are shown for Crew 1 (baseline case) and Crew 2 (uplink case). As planned, the cross-tasking of crew exposure to the two weather scenarios ensured that the crews were not presented with the same scenario twice. We were able to see some variation in crew interpretation and response using this strategy. Figure 8, with visual inspection only, appears to show the uplink CTOP information actually caused a larger deviation than the baseline case. Most of the flight tracks coincide up to the point where the weather case required deviation. A significant observation is that (via inspection only) the uplink CTOP (Crew 2) resulted in a deviation that missed most of the large, intense convection, whereas the baseline (airborne radar, Crew 1) information resulted in a flight track that penetrated an area of widespread, intense convection. This one case is merely an observation, but it illustrates one potential benefit of frequent CTOP updates.

24

Figure 9 is the 27 August 2010 case, with Crew 1 having the benefit of CTOP uplinks and Crew 2 using airborne radar and AOC guidance only. Here we observe another potential benefit of frequent updates—the deviation is much less for the Crew 1 flight track than Crew 2’s baseline flight. Again, this is an observation only. However, it can be argued that Crew 1 penetrated a larger area of intense convection. The observer noted for this case that the CTOP update information was used to look-ahead, thereby adding confidence that the convection could be avoided tactically using the airborne radar without potential attenuation blanking further out.

An important observation through pilot reaction and real-time comments was that the pilots became more adept at the proper use of the CTOP updates as they became more experienced through exposure to the scenarios and information. That is, the uplink update is more properly used as a strategic tool that supplements the airborne radar, which remains the primary source of information when/if faced with the need for tactical avoidance.

Again, the reader is referenced to Appendix F for a waypoint breakdown of these two cases along with the actual color graphic CTOP display for each segment.

To summarize other pilot comments, it was felt that a larger CTOP view down track would have decreased the number of deviation requests. Scenario #2 presented a weather scenario where knowledge of what was beyond the initial storm complex would have supported a turn that would have decreased the miles off-course.

3.1.4. Subjective Safety Enhancement Pilots rated enhanced safety as high when given the updated CTOP information with comments like

• “Excellent situational awareness tool” • “Obvious, can assist in long range planning, avoiding short range weather avoidance” • “Great help for pilots…” • “Results in more meaningful discussions with dispatch” • “Very useful as long as the data is valid”

The last bullet was captured in a written questionnaire response. Observing the pilots actively diverting around weather hazards helped to clarify this conditional statement to include congruence with the airborne weather radar display. Pilots depend heavily on the ND radar information, and it is in fact the primary source of information on both convection and turbulence. Future airborne demonstrations must ensure some comparability with the airborne weather radar, to include similar color mapping to severity, comparable ranges and scaling, and relationship to the actual flight profile. Pilots struggled at times when the uplinked information scaling differed from the ND, and information transfer for decision support did not happen easily. Some answers are contained in Section 3.5; others will become apparent when the methods for display and user manipulation for future demonstrations become known.

25

Pilots universally agreed that proper training on the final display configuration and weather product limitations was essential. The pilots felt that the limited exposure to the enhanced weather information and displays they received during training was not enough to make them proficient on use until the actual demonstration run. Initially, there was “on-the-job training” happening.

The bottom line is to avoid presenting anything to the pilot while in-flight that might be in conflict with the other information the pilot has. “In conflict” also means interpretation or perception that result in more uncertainty.

3.2. Human Factors Measures

3.2.1. Pilot Workload Subjective comments from the pilots indicate that pilot workload was increased, but it was not detrimental to safely completing the flight. Any time additional information is given during a task, the effort to create strategies for its use will increase workload. However, in the case of oceanic flights, pilots tend to anticipate the next update and with time understand how the information is best used for decision support. The updates were missed during baseline scenarios, and were readily used to plan for the weather encounter in the enhanced cases. The idle time between waypoints was used by the pilots to strategize and confirm potential courses of action with the AOC.

Ratings for tasks such as monitoring weather along your route-of-flight; making reroute decisions; maintaining awareness of the flight environment—the pilots universally agreed that enhanced weather information is much more effective. In this case, the increased workload actually enhanced pilot awareness of the situation.

3.2.2. Controller Workload The effect of enhanced weather information on controller workload is difficult to generalize. For this trip scenario, ATC was used to confirm turn direction to initiate the deviation. Once the deviation strategy was determined and implemented, ATC was not used except for ride report requests and position reporting. This scenario assumed that ATC had access to limited weather radar data plus the scenario PIB. In previous demonstrations in the Pacific, ATC had little decision support information, and so workload relative to information requests was minimal.

To somewhat generalize the effect on ATC workload, since strategic deviation was possible with the enhanced CTOP information, the reduction in deviation requests should decrease controller workload.

3.2.3. Airline Operations Center Workload During the baseline flights, the interaction between the pilot and AOC was more oriented toward obtaining information with little or no awareness of what was ahead (other than the information contained in the PIB). For the enhanced case, the inquiries were always prompted by either or

26

both an ND and/or the CTOP information display. The interaction tended to start with the pilot describing the situation and asking for guidance on which way to turn, left or right. Further, in the enhanced cases, the AOC guidance agreed with the pilot’s initial assessment of the required decision.

Pilots suggested on the questionnaire that the CTOP might encourage more AOC/pilot interaction. However, the interaction was more of a discussion than a request, and the quality of the discussion in terms of decision support was improved. The conclusion was AOC workload was not decreased with the addition of a weather update capability for the pilot, but the interaction approached a state of shared situational awareness with both pilot and dispatcher using similar information. The CTOP information provided the stimulus for the interaction vs. the airborne radar, simply because of the operational range limitations of the airborne radar (160nm effective, 320nm nominal).

3.2.4. Situational Awareness Discussion Pilots universally rated general situational awareness as much more effective with the enhanced CTOP information. Logically, it flows that decision support for reroutes and deviations is much more effective in the enhanced case. There were comments from the pilots that certain characteristics of the presentation of information needed change in order to realize the full benefit. So, separating the information content from the information transfer mechanism—the display concept and pilot-machine interface—we can conclude that the information is effective, but how that information is presented to the pilot needs some work. Specifically (Section 3.5):

• Increased look-ahead distance • Range rings or tick marks that are similar in scale to the ND radar • Ownship positioning and relative track to the planned course • More vertical layers

Anecdotally, the strategy one pilot chose during the baseline scenario was arbitrary—“might as well go right”—when the weather first appeared on the ND. The result was an eventual deviation of 110nm off course. Tactical “hunt and peck” when 40-80nm from the leading edge of the weather continued through the entire weather case. A pilot from crew 2 flying the same scenario with enhanced CTOP information turned left well away from the leading edge of the weather and avoided any tactical maneuvering. This was a very good example of enhanced awareness of the environment even with the display limitations described above.

3.3. Observer Ratings Observers included a retired military pilot with extensive oceanic experience and a project scientist responsible for the development of convective weather products for continental as well as oceanic regions. Observer ratings were entirely consistent with the pilots’ input and will not be repeated here. An attempt was made to describe decision-making and pilot comments of each scenario, and correlating these with known unique characteristics of each weather case.

27

One example of an enhanced CTOP decision strategy is given here. The new CTOP display uplinked to the RCS gave pilots the indication that after abeam DAGUD (a waypoint) the pilot could turn right back on course without fear of running into attenuated cells. The character graphic shows many more convective cells left, but since they turned right to direct BUXOS (the next waypoint), they avoided these cells. In summary, there was minimal deviation from course vs. 120nm deviation by the other crew in baseline case.

Another example strategy used when CTOP graphics became available was the initial use

of small heading changes initially that appeared to avoid the weather complex, and then adjusting those small deviations as the weather showed up on the ND. The goal was to minimize the total deviation from the planned track. The strategy was successful, resulting in a maximum course deviation of 4.7nm during the first half of the weather case, then maintaining separation from the weather by paralleling the planned track 4.0nm right. The strategy also illustrated a shortcoming of the CTOP update strategy which prompted pilot suggestions of more frequent updates—as the CTOP aged, it became more difficult to correlate with the ND radar display. Therefore, for this case, the CTOP update was useful for selecting the initial avoidance strategy but decreased in usefulness as time went on.

3.4. Support Tool Effectiveness

3.4.1. Preflight Information Bulletin Pilots were provided typical PIB materials prior to each flight scenario. The package included relevant SIGMETs, satellite imagery, and weather outlook for the CONUS. The information was extracted from NCAR archives to coincide with the rest of the weather scenario. It did not include NOTAMs.

The pilots were observed to review the PIB materials together prior to flight and to discuss the overall weather picture for the route. The information set the stage for what could be expected—it provided a big picture view and set expectations for what to expect on the ND. The PIB used for this demonstration was very useful and served its purpose.

3.4.2. Airborne Weather Radar The airborne weather display on the ND that was used for the demonstration was powered by an NCAR-developed radar simulator that was in turn driven by actual weather data for the chosen scenario. The information presented to the pilots was very effective and realistic, including a routine that simulated potential attenuation. The pilots depended on the ND information for tactical avoidance and, when within 320nm of the weather, a strategic view of what was coming along the planned flight track. Pilots were able to change radar range throughout the Airbus-provided range selections—from 40nm to 320nm.

There were two limitations that were noted by the pilots. First, the radar simulator and RCS were not able to portray the effects of changing radar tilt, so the pilots were only provided a

28

zero degree tilt slice of the weather. It was noted that the tilt feature is an important capability for characterizing the vertical structure of the storm complex. Second, the radar did not attempt to model the real-world range limitations noted by the pilots—that is, the ND radar display always provided a true return all the way out to the nominal range of the radar, 320nm. In “real world” situations, the pilots noted that the width of the beam at 320nm range selection becomes wide enough to distort most weather painted at that range.

3.4.3. Electronic Flight Bag The electronic flight bag (EFB) was a new and unique feature of this demonstration. It was modeled after a Class 2-type EFB provided by a major supplier of EFB for airline operations. For this demonstration, the only capability activated was the CTOP color graphic and character graphic displays described earlier. After each ACARS position report, the CTOP product simulator generated a CTOP product and both graphics were displayed within a few minutes of the position report. Aircraft position was displayed by a red “X” at the time that the ACARS sent the position report. Pilots used the MCDU to send the report.

The EFB performed flawlessly for the demonstration, and was used extensively by the pilots. It allowed the pilot to toggle at-will between the color and character graphic. During shakedown, it was noted that the character graphic might be difficult to read given the aspect ratio needed to retain relative trueness, so pilots were also given a paper rendition of the character graphic for reference. However, in most cases the pilots were able to use the character graphic as displayed on the EFB. One nice feature of the EFB that was not anticipated was the ability for the pilots to select previous CTOP products. They frequently referred back to previous CTOP graphics to understand the movement and progression of the weather complex.

Pilots universally used the CTOP to determine the potential for attenuation on the ND radar depiction. More vertical resolution was desired than that given in order to better determine if the flight could overfly storm areas—airlines typically require 5000 feet overflight separation from cells. The 30-40,000 foot slice was too large to determine whether vertical separation standards could be maintained.

3.4.4. Enhanced Weather Information (EFB) Several pilot comments and decisions that illustrate the effectiveness of the enhanced weather information display are repeated below:

• Pilot verbal feedback on the ASCII display was mostly positive, a unique way of conveying information without using link bandwidth or re-equipage. One pilot commented “pretty nice.”

• Based on ND radar alone, pilots were tempted to “thread the needle” through the storm areas; however, the CTOP indicated the potential for attenuated returns behind the initial line of storms.

29

• Pilots developed (and became proficient with) strategies that involved many small heading changes using the CTOP display for guidance, then supplementing these initial deviations with radar when the storms came into view. This minimized the total deviation from course.

• One pilot commented that after being exposed to the CTOP display during training, he really missed not having it during the baseline case.

• Many times the pilots were able to begin to get back on course as soon as possible given the look-ahead provided by the CTOP.

• Pilots constantly referenced their use of CTOP to identify potential attenuation. They were constantly cross-referencing the ND with the EFB display while attempting to determine the best strategy.

Pilots did not identify any safety concerns with the CTOP display, either color or character graphic. They did identify some enhancements that might be enabled by the progression of more capable EFBs onto the flight deck (such as tablet computers). See Section 3.5.

3.4.5 RCS Out-the window (OTW) View (Figure 8) “One peek (out the window) is worth a thousand cross-checks (on instruments).” The RCS out-the-window view of the individual cells turned out to be of value when the pilots were devising a deviation strategy or even during tactical maneuvering. This was true even during full night operations because of the lightning flashes and resulting illumination of individual cells. The OTW capability needs to be further refined and become a core capability for the RCS. One issue of realism was noted—pilots commented on the fact that most of the time individual cells were embedded and sometimes hidden by clouds. This did not diminish the dependence pilots have of a look out the window to verify what is shown on the ND radar and CTOP displays.

30

Figure 10. A photo of the First Officer’s ND and OTW view.

3.5. Discussion of Support Tool Improvements Once familiar with the limitations and information content of the EFB CTOP displays, pilots were generally enthused by the new capability. Ideas for future improvement to the updated weather information and display concepts that were presented included:

• The EFB should have some of the Class 3 EFB capabilities, such as own-ship position with track up that shows real-time aircraft position and track information relative to the weather displays. The current system where aircraft position is only shown when weather is updated sometimes required pilot “mental gymnastics” to relate current aircraft position to the weather hazard.

• The EFB display should be oriented similar to the 320nm range of the airborne radar, and include distance from current position to weather using similar range rings. Again, the pilots were required to mentally reconstruct the EFB image so that it was congruent to the airborne weather radar ND.

• The CTOP display should have a look-ahead range greater than the distance out to the furthest waypoint. Another problem that needs future attention is the ASCII image (in most cases) extended to the third waypoint, and the color graphic (again, in most cases) showed CTOP information beyond the third waypoint. Further pilot input and study should converge on a desired range for future demonstrations. One pilot suggested a request-reply concept which would include maximum look-ahead distance. Pilots suggested a 500-1000nm look-ahead; one pilot suggested a look-ahead all-the-way to destination.

• Pilots thought that a “push” of weather information should occur whenever the satellite data update.

• The wind vector is important in deciding which way to deviate around a cell. Pilots universally chose the upwind side of the storm area, even if it resulted in a larger

31

deviation. Airlines generally require a 20nm separation from cells above FL250, and a 10nm separation below FL250.

• Future demonstrations should consider adding more vertical layers, especially between FL300-400. This is where most oceanic flights operate. One pilot suggested another level at FL500, similar to the AF447 weather case.

• Spirit Airlines Airbus radars add a magenta “band” to the display whenever attenuation is probable. Pilots missed this piece of information on the RCS ND.

• A nice feature of the EFB CTOP for this demonstration was the ability to recall previous CTOP images, allowing pilots to view historical storm cell trends.

• A few pilot comments on the ASCII display indicate there may be an opportunity to further refine it prior to future demonstrations. There was a concern with clutter that might be addressed by using a character other than the “slash” and eliminating quotes around characters defined in the legend. The CTOP product generating system used for the HOTL demonstration was the same as

that used in previous demonstrations with United Airlines B-777s and B-747s in the Pacific. The generation of CTOP products for uplink was triggered by ACARS position reports every 10 degrees of longitude. The pilots who participated in this HOTL demonstration emphasized that ADS-B is used by most aircraft now, which results in position reporting every five minutes. Future demonstrations will most likely be in this environment and will require a different triggering mechanism. Also, as airlines equip with Class 2 and 3 EFBs, the character graphic may not be the preferred display method. Color graphics with zoom capability similar to Apple and Android-type tablet displays may be the most likely display concept for future demonstrations.

Pilots universally identified other oceanic weather hazards as potential enhancements to

an oceanic/remote weather update capability. These are (in priority order): • Turbulence, clear-air and convective-induced • Wind streamlines or vectors • Volcanic ash • Icing exceeding aircraft limits for ETOP planning and diversion

4. CONCLUSIONS The overall conclusion reached by the HOTL team was the demonstration was extremely valuable in establishing the merits of oceanic weather updates. The comments provided by the demonstration pilots were outstanding, both positive and constructively criticizing. Many suggested enhancements were identified that will increase the value of future demonstrations on actual line trips. It was also universally understood that a full slate of weather hazards—not just convection—is needed as stage lengths over-water exceed 15 hours.

Specific recommendations are noted as a result of this demonstration:

32

• Additional research and product development are justified by the potential safety and efficiency enhancements resulting from cockpit update of weather hazards, especially for oceanic flights but also for long trans-continental flights.

• A seamless transition from continental to oceanic weather updating is needed as flights depart from other than coastal gateways in the U.S.

• The next step is to prepare for and accomplish weather uplink to actual line trips, making use of whatever infrastructure is available without re-equipage. Validating the science and usability of advanced weather products can only occur if the users experience the technology and are able to provide operational feedback to researchers.

• The next step must include the capability to use advanced user interfaces as they are introduced to line operations. The ASCII character graphic is a basic step to get the information to the flight deck. As fully integrated EFBs (as well as tethered tablets) are introduced, and broadband Internet becomes available on aircraft, the future demonstrations need to utilize that enhanced capability.

• Flight crew training on devices and weather product limits and capabilities must precede any future demonstrations.

A phased approach for future work is proposed, given these conclusions:

Phase 1. The first phase of a multi-year plan for a national oceanic weather capability is the WTIC Transoceanic Human Over the Loop (HOTL) demonstration at the FAA WJHTC NIEC. This phase will be complete, objectives satisfied, when the results are collectively accepted by the HOTL NIEC Team.

Phases 2-6 (based on Phase 1 results and direction from the FAA AWO).

The capability to produce the cloud top height product and uplink it into the cockpit still exists at NCAR. Reinvigoration of this capability would entail the following steps:

1. Commitment from the FAA to fund this reinvigoration and to support a possibly long-term (5-10 years), experimental demonstration with interested airlines.

2. FAA specification of oceanic dispatcher requirements for viewing the products prior to uplinking so as to have a common situational awareness with the pilots. Purchases of hardware and/or internet service provider access may be required and should be funded by the FAA. Initially, working with the Oakland Center for Pacific oceanic flights can be done. Expansion to Houston and New York centers for Gulf of Mexico/Caribbean/South American and for north Atlantic flights, respectively, can be added during a phased implementation.

3. Purchase of a subscription to an international waypoint location service will be funded by the FAA. The ARINC communications link currently active at NCAR which is directly funded by the FAA AWRP, can be utilized for this demonstration (costs of this communication link are not included here).

33

4. Testing of the NCAR uplinked product creation and uplink capability is needed to ensure the system remains functional from its termination after the successful 2006 demonstration. ARINC software responsible for ACARS communication must be regenerated.

5. Pilot briefing prior to uplink demonstration is to be coordinated between NCAR and participating airlines. NCAR will prepare material for Flight Crew Bulletins for use by the airlines.

6. NCAR will operate the software for uplinking the cloud top height product into participating aircraft. However, this experimental system cannot be guaranteed to run 24x7.

7. Pilot and dispatcher feedback will be solicited via a web page and by written comments to ensure that the product is meeting inflight and dispatcher needs for weather hazard information. A feedback questionnaire will be provided, with AWRP Quality Assessment PDT assistance, for use by the pilots and dispatchers. Quality Assessment PDT can provide summary reports on the effectiveness and utility of the uplinked products.

8. Improvements/modifications to the cloud top height display will be considered to include additional information such as global model-derived turbulence intensity estimates. Collaboration with the participating airlines will guide any improvements/modifications.

9. Initially, only regions contained with the GOES-East and GOES-West satellite domains will have the capability for an uplinked weather product. Expansion of the program to include coverage of regions served by the Meteosat-9 (Europe and Africa) and the MTSAT-1 (Japan and Australia) satellites is possible, provided that real-time access to the data streams can be achieved. Access and development costs are anticipated but unknown at this time.

10. Possible path to operations can be explored by including companies like Boeing and other Electronic Flight Bag (EFB) providers to leverage costs between Government and industry. Aircraft implementation is not included in this proposal, but rather working with industry is proposed to ensure uplink data formats can support displays like the color graphic shown in the right panel of Figure 1. The color graphic display was originally designed to be shown on a Boeing 777 aircraft EFB screen. American Airlines now uses the EFB screen as an ACARS display and means no re-equipage is required to show a more compelling graphic.

The above items contain the steps necessary for a full implementation plan. A phased approach could be utilized to good advantage.

Phase 2.

Includes items 1-7 above for a demonstration with United Airlines only as this airline hosted the previous uplink demonstration for their Pacific oceanic flights. The Oakland dispatch center will have their previous web-based display capability installed. Items 1-7 allow prior capabilities to be re-established and made functional and provide opportunities for pilot evaluation/feedback.

Phase 3.

34

Expansion of United Airlines oceanic uplink coverage to include those flights into the Gulf of Mexico/Caribbean/South America regions and to Atlantic flights to Europe/Africa. Only regions covered by the GOES-East satellite are included. Coordination with the Houston and New York dispatch centers for installation of web-based display of the uplink product will be done.

Phase 4.

The addition of item 8 to make enhancements to the ASCII uplink display by adding turbulence information for the United Airlines demonstration only.

Phase 5.

Additional airline participation can be added as needed, assuming their flights are within GOES satellite coverage and that they have similar inflight equipages as is used by United Airlines. This estimate includes development, pilot training, equipment and travel. Completion of this task is expected to take 5-6 months per airline.

Phase 6.

Add uplink product coverage using imagery from the Meteosat-9 and MTSAT-1 satellites (Item 9).

35

References

Donovan, M., E. Williams, C. Kessinger, G. Blackburn, P. H. Herzegh, R. L. Bankert, S. D. Miller, and F. R. Mosher, 2008: The identification and verification of hazardous convective cells over oceans using visible and infrared satellite observations. Journal of Applied Meteorology and Climatology, 47, 164-184.

Federal Aviation Administration. (2010). FAA’s NextGen Implementation Plan 2012. Retrieved from http://www.faa.gov/nextgen/implementation/

Herzegh, P.H., E.R. Williams, T.A. Lindholm, F.R. Mosher, C. Kessinger, R. Sharman, J.D. Hawkins, and D.B. Johnson, 2002: Development of automated aviation weather products for oceanic/remote regions: Scientific and practical challenges, research strategies, and first steps. Preprints, 10th Conference on Aviation, Range and Aerospace Meteorology, American Meteorological Society, Portland, 13-16 May 2002.

Joint Planning and Development Office (2009). Concept of Operations for the Next Generation Air Transport System, v3.0 (October 1, 2009). Retrieved from www.jpdo.gov/library/NextGen_ConOps_v3.2.pdf

Kessinger, C., M. Donovan*, R. Bankert*, E. Williams*, J. Hawkins*, H. Cai, N. Rehak, D. Megenhardt, and M. Steiner, 2008: “Convection diagnosis and nowcasting for oceanic aviation applications” in Remote Sensing Applications for Aviation Weather Hazard Detection and Decision Support, edited by Wayne F. Feltz, John J. Murray, Proceedings of SPIE Vol. 7088 (SPIE, Bellingham, WA, 2008) 7088-08, San Diego, 10-14 August 2008.

Miller, S., T. Tsui, G. Blackburn, C. Kessinger, and P. Herzegh, 2005: “Technical Description of the Cloud Top Height (CTOP) Product, the First Component of the Convective Diagnosis Oceanic (CDO) Product”. Submitted to the FAA AWRP on 11 March 2005 as a requirement for the CTOP AWTT D3 decision scheduled for May 2005.

36

Acronyms

ACARS Aircraft Condition and Reporting System ANSP Air Navigation Service Provider AOC Airline Operations Center ATC Air Traffic Control (or Controller) ATM Air Traffic Management (or Manager) CDO Convective Diagnostic Oceanic CONUS Continental U.S. CRP Compulsory Reporting Point CTOP Cloud Top Height EFB Electronic Flight Bag FAA Federal Aviation Administration FANS Future Air Navigation System FCB Flight Crew Bulletin FMS Flight Management System HITL Human-in-the-loop HOTL Human-over-the-loop ICAO International Civil Aviation Organization ITCZ Intertropical Convergence Zone LOFT Line-oriented Flight Training MCDU Multi-function Control and Display Unit NCAR National Center for Atmospheric Research ND Navigation Display NIEC NextGen Integration and Evaluation Capability OPSPEC Operational Specification PDT Product Development Team PIB Preflight Information Bulletin RAL Research Applications Laboratory RCS Research Cockpit Simulator SME Subject Matter Experts UTC Universal Coordinated Time WJHTC William J. Hughes Technical Center WTIC Weather Technology in the Cockpit

37

Appendix A – Informed Consent Statement

I,___________________________________________, understand that this study, entitled "Weather Technology in the Cockpit (WTIC) Transoceanic Human-Over-the-Loop (HOTL) Demonstration,” is sponsored by the Federal Aviation Administration (FAA) WTIC Program Office and is being directed by Tenny Lindholm and Cathy Kessinger from the National Center for Atmospheric Research (NCAR).

Nature and Purpose: I have been recruited to volunteer as a participant in this project. The purposes of the study include:

1. Evaluating the risk of in-flight evaluations of updated weather information in oceanic/remote regions,

2. Increasing our understanding of impacts to pilot, dispatch, and air traffic management (ATM) decision making in a collaborative environment when updated oceanic weather information is provided to the flight deck, and.

3. Identifying demonstration objectives that are best accomplished with an expanded demonstration of uplinked hazardous weather information to transoceanic airline flights.

Experimental Procedures: Two flight crews (four pilots) will be selected based on route and aircraft experience. Crew #1 will arrive at the WJHTC NextGen Integration and Evaluation Capability (NIEC) on Monday, 16 July 2012, to receive training on the Research Cockpit Simulator. Crew #1 will then fly two scenarios from Ft. Lauderdale, FL, to Lima, Peru, on Tuesday and Wednesday. Crew #2 will arrive at the WJHTC NIEC RCS facility for training on Wednesday, 18 July 2012. The crew will then fly the same two scenarios with identical weather cases on Thursday and Friday. The demonstration environment will simulate an actual Airbus A-320 trip with Air Traffic Control and Airline Operations Center roles being played by respective SMEs. Each flight scenario will be four hours in duration. Each flight period will include a one-hour pre-brief and post-trip debrief. Weather scenarios have been developed from actual hazardous convective weather cases. For the Baseline Case, weather information provided to the pilots will simulate what is done currently—paper text SIGMETs, paper satellite imagery, and data linked textual SIGMETs and warnings (via ACARS). The pilots will also have a simulated airborne weather radar depiction on their navigation displays. For the Enhanced Case, convective cloud-top height products will be generated and updated at each position report transmission from the flight. A character graphic plus a color graphic will be provided on the simulated Electronic Flight Bag. Changes in decision-making and overall situational awareness will be inferred, observed, or elicited from the pilots during the flight. Weather generation for the Enhanced Case will occur at NCAR in response to an electronic position report received from the flight. NCAR, FAA and WJHTC will observe and have available an Observer Form to guide the collection of qualitative data from the pilots. Since there will be only two demonstrations for each scenario, no statistical findings can be made.

Discomfort and Risks: I understand that during this study, I will not be exposed to any foreseeable risks.

38

Confidentiality: My participation is strictly confidential, and no individual names or identities will be recorded or released in any reports. Qualitative findings will be reported to the Sponsor without any reference to pilot identity.

Benefits: I understand that the only benefit to me is that I will be able to provide the researchers with valuable feedback and insight into the automation required to support future separation management. Data originating from this study, will aid the FAA in identifying human factors issues associated with i future transoceanic demonstrations, planned for actual revenue flights. I also understand that I may benefit from enhanced knowledge of future weather diagnoses planned for the oceanic flight regime.

Participant Responsibilities: I am aware that to participate in this study I must be a current and qualified pilot with a U.S. air carrier. I will perform my cockpit duties and answer questions to the best of my abilities. I will not discuss the content of the experiment with anyone until the study is completed.

Participant's Assurances: I understand that my participation in this study is completely voluntary, and I have the freedom to withdraw at any time without penalty. I also understand that the researchers in this study may terminate my participation if they feel this to be in my best interest. I understand that if new findings develop during the course of this research that may relate to my decision to continue participation, I will be informed.

I have not given up any of my legal rights or released any individual or institution from liability for negligence.

The FAA Sponsor has adequately answered all the questions I have asked about this study, my participation, and the procedures involved. I understand that NCAR, the FAA, or another member of the research team will be available to answer any questions concerning procedures throughout this study.

If I have questions about this study or need to report any adverse effects from the research procedures, I will contact Mr. Tenny Lindholm, (303) 497-8448, email lindholm@ucar.edu.

Compensation and Injury: I agree to immediately report any injury or suspected adverse effect to the sponsor, Mr. Ian Johnson, FAA ANG-C61, (202) 385-7168.

Signature Lines: I have read this informed consent statement. I understand its contents, and I freely consent to participate in this study under the conditions described. I understand that, if I want to, I may have a copy of this statement. Research Participant:______________________________ Date______________

Investigator: _______________________________________ Date______________

39

Witness:_________________________________________ Date______________

40

Appendix B – Pre-evaluation Questionnaires

Pre-demonstration Questionnaire

Flight hours, total/in-type

Experience on oceanic routes yes/no

What weather information is normally provided prior to departure on oceanic routes?

What type of weather information is available to you from Dispatch while on oceanic routes?

Do you have any other sources of en route weather information? If so, what are they?

What are your airline policies regarding avoidance of hazardous convective weather? That is, vertical and horizontal separation requirements given an airborne weather depiction of color (red, yellow, green).

41

Appendix C – Computer Flight Plan and Scenario Script

Computer Flight Plan Alternates CIX or TRU. Only one route used unless a reroute is needed for weather/volcanic activity etc. // SPIRIT AIRLINES FLIGHT PLAN FLT 0320 // ORIG FLL/KFLL FT LAUDERDALE HOLLWOOD IN N26044W080092 SPCL AP DSTN LIM/SPIM LIMA S12013W077069 ALTN CIX/SPHI CHICLAYO-J A GONZALES S06472W079497 SCHEDULE ACTUAL OUT: 2055Z ----------------------------------------- IN: 0235Z ----------------------------------------- COMPUTED 05MAY1932Z FOR ETD 2230Z PROGS 0512 CRZ:M78 IFR REL NKS0320 /05 KFLL/SPIM IFR A319-132 N505NK PERF FCTR 03.9 AVG AVG AVG FUEL TIME CORR CLB CRZ DSC DEST SPIM 029515 0520 . . . . PLANNED WC P003/P005/P001 RESV 002587 0032 . . . . BOW 091825 TD P012/P009/P014 ALTN SPHI 005073 0053 . . . . PAYLOAD 026681 HOLD 002419 0030 . . . . ZFW 118506 MAX 128969 CONT 000000 0000 . . . . RQD AT TO 039594 .... . . . . TO FUEL 039594 EXTRA 000000 0000 . . . . TOGWT 158100 MAX 158100 TAXI 000453 0017 . . . . BURN 029515 TTL RAMP 040047 0732 . . . . LDGWT 128585 MAX 137787 FOD 010079 T/O ALT NIL 1ST ALT SPHI DIST 0352 W/C M004 SPIM DCT SLS TRU DCT SPHI BLOCK TIMES FLIGHT TIMES FUEL IN . . . . ON . . . . BLOCK OUT . . . . OUT . . . . OFF . . . . TAKEOFF . . . . TOTAL . . . . TOTAL . . . . BLOCK IN . . . . BURNOFF . . . . BURN TIME FL PROFILE PLAN 029515 05:20 FLL 360 URSUS 350 GYV 370 TRU 390 DOMST BONDED FLL FUEL COST 3.36 3.34 LIM FUEL COST 3.54 3.54 TANKER SAVINGS 0.00 INCREMENTAL BURN PER 1000 LBS INCREASE IN TOW: 168 KFLL BEECH2 BAHMA URSUS UL780 TRU UG436 LIM DCT SPIM

42

WAYPOINT AIRWAY M/H DIST ZTME WIND TMP WSH W/C ZBRN FUEL TO FL M/C ACDST ACTM ETA/ATA TAS G/S PFR AFR VAR --------------------------.----------.-------.------------.-----.---- TAKEOFF FUEL PAD: 0000 LANDING FUEL PAD 0312 --------------------------.----------.-------.------------.-----.---- N25565W079078 BEECH2 102 56 0016 344008 CL 01 P003 0019 .. BAHMA CL 103 56 0016 .../... 296 299 0377 .... .... --------------------------.----------.-------.------------.-----.---- N24076W079044 DCT 187 109 0014 329034 CL 02 P029 0023 .. TOC CL 184 164 0030 .../... 440 470 0354 .... .... --------------------------.----------.-------.------------.-----.---- N24000W079042 DCT 188 8 0001 309036 M51 02 P022 0001 .. URSUS 360 184 172 0031 .../... 455 478 0353 .... .... --------------------------.----------.-------.------------.-----.---- N21188W079078 UL780 192 161 0020 297040 M46 02 P016 0020 .. PIGBO 350 187 333 0051 .../... 461 477 0333 .... .... --------------------------.----------.-------.------------.-----.---- N20000W079095 UL780 192 79 0010 291043 M45 02 P013 0010 .. GAXER 350 186 411 0101 .../... 462 475 0324 .... .... --------------------------.----------.-------.------------.-----.---- N15000W079197 UL780 192 299 0038 282044 M43 02 P005 0037 .. DAGUD 350 186 710 0140 .../... 463 469 0287 .... .... --------------------------.----------.-------.------------.-----.---- N05101W079400 UL780 187 587 0116 262009 M43 01 M002 0071 .. BUXOS 350 186 1297 0256 .../... 464 462 0216 .... .... --------------------------.----------.-------.------------.-----.---- N01250W079500 UL780 184 224 0029 043010 M43 03 P008 0026 .. UGUPI 350 185 1521 0324 .../... 464 471 0190 .... .... --------------------------.----------.-------.------------.-----.---- S02077W079520 UL780 181 212 0027 047007 M43 03 P005 0024 .. GYV 115.9 350 182 1733 0352 .../... 462 467 0166 .... .... --------------------------.----------.-------.------------.-----.---- S04302W079340 UL780 174 143 0019 269003 M49 01 P000 0017 .. VAKUD 370 173 1875 0410 .../... 458 458 0150 .... .... --------------------------.----------.-------.------------.-----.---- S08052W079067 UL780 174 216 0028 276013 M50 01 P003 0024 .. TRU 116.3 370 172 2091 0438 .../... 458 461 0126 .... .... --------------------------.----------.-------.------------.-----.---- S09089W078313 UG436 152 72 0009 277017 M53 00 P010 0008 .. BTE 112.5 390 150 2163 0448 .../... 455 465 0118 .... .... --------------------------.----------.-------.------------.-----.---- S10118W078006 UG436 155 70 0009 283018 M56 00 P011 0007 .. ATOGO 390 153 2233 0457 .../... 451 462 0110 .... .... --------------------------.----------.-------.------------.-----.---- S10357W077489 UG436 156 26 0003 286018 DC 00 P012 0003 .. TOD DC 154 2259 0500 .../... 451 463 0107 .... .... --------------------------.----------.-------.------------.-----.---- S11108W077317 UG436 156 39 0006 288015 DC 00 P010 0001 .. GALGO DC 154 2298 0506 .../... 386 396 0106 .... .... --------------------------.----------.-------.------------.-----.---- S12005W077074 UG436 153 55 0009 088011 DC 01 M004 0002 .. LIM 113.8 DC 154 2353 0515 .../... 386 382 0104 .... .... --------------------------.----------.-------.------------.-----.----

43

S12013W077069 DCT 148 1 0005 129000 DC 00 P000 0003 .. SPIM DC 148 2354 0520 .../... 386 386 0101 .... ....

- - - - - - - - - - - - - ALTERNATE SPHI - - - - - - - - - - - - - - SPIM DCT SLS TRU DCT SPHI --------------------------.----------.-------.------------.-----.---- S11173W077338 DCT 331 51 0010 077008 CL 01 P002 0016 .. SLS 114.7 CL 329 51 0010 .../... 310 312 0059 .... .... --------------------------.----------.-------.------------.-----.---- S10084W078074 DCT 332 76 0011 285016 CL 01 M010 0017 .. TOC CL 333 127 0021 .../... 443 433 0042 .... .... --------------------------.----------.-------.------------.-----.---- S09164W078325 DCT 332 134 0018 281018 DC 01 M011 0014 .. TOD DC 333 261 0039 .../... 451 440 0028 .... .... --------------------------.----------.-------.------------.-----.---- S08052W079067 DCT 332 2 0000 285018 DC 01 M012 0000 .. TRU 116.3 DC 333 263 0039 .../... 384 372 0027 .... .... --------------------------.----------.-------.------------.-----.---- S06472W079497 DCT 333 89 0014 079014 DC 01 P004 0003 .. SPHI DC 330 352 0053 .../... 384 388 0024 .... .... POINT FL100 FL220 FL330 FL350 FL370 FL390 KFLL 121/003 340/023 329/032/M44 324/032 M48 315/035/M53 308/041/M56 BAHMA 092/007 328/024 309/033/M43 303/037 M48 297/042/M51 294/048/M54 URSUS 043/005 318/022 298/040/M42 291/043 M45 286/044/M50 283/047/M53 PIGBO 357/006 314/022 292/037/M41 290/043 M45 287/050/M49 284/056/M52 GAXER 353/006 293/016 270/035/M38 269/037 M43 266/038/M49 263/036/M54 DAGUD 299/005 128/012 076/010/M38 043/008 M43 026/009/M49 021/009/M55 BUXOS 358/007 112/019 065/010/M38 040/012 M43 036/010/M49 048/007/M55 UGUPI 099/005 093/019 107/005/M38 121/002 M44 049/000/M49 294/002/M55 GYV 094/009 090/021 317/003/M38 294/004 M44 279/007/M50 268/009/M55 VAKUD 078/003 085/016 283/012/M39 277/016 M44 275/017/M50 275/017/M55 TRU 104/003 082/013 287/013/M40 278/016 M45 279/018/M50 281/018/M56 BTE 139/005 071/013 291/014/M40 283/017 M45 283/018/M51 285/018/M56 ATOGO 134/008 062/012 288/015/M40 282/017 M45 284/018/M51 287/019/M56 GALGO 145/009 054/010 287/016/M40 284/018 M45 283/019/M51 286/019/M56 LIM 145/009 054/010 287/016/M40 284/018 M45 283/019/M51 286/020/M56

44

Scenario Script Weather Technology in the Cockpit (WTIC)

Transoceanic Human-Over-the-Loop (HOTL) Demonstration FAA Next Generation Integration and Evaluation Capability (NIEC) Research Cockpit Simulator

(RCS)

William J. Hughes Technical Center (WJHTC) Atlantic City, NJ

The demonstration will be completed in two simulator sessions for each flight crew (four sessions total). The first flight (for each crew) will be a baseline case; the second will avail enhanced weather information that is presented on the EFB. Sessions will be four hours long. Weather case events will be drawn from actual regional weather scenarios that would be expected to elicit flight crew responses (that is, requests for more information, deviation, or both).

For each flight crew, a scenario will be presented with weather information as currently available, and then with updated weather hazard information on a graphical display (ACARS character graphic and color graphic on the EFB). Decision making behavior and feedback from pilots will be recorded and used to establish a concept of use baseline for Operational Demonstrations on actual oceanic flights.

Data Collection

Data collected and documented for the final report will include:

1. Pilot input on what other weather hazard graphic depictions, plus functionality, should be added to the EFB for further evaluations (for example, vertical cross-section, turbulence, jet stream). This feedback from line pilots is important and will be solicited both during and after each simulator session.

2. Pilot voice and/or data communications (ACARS) with dispatch and ATC (number of pilot initiated requests for information, NCAR).

3. Recording of actual deviations around weather that are initiated by the pilot without ATC clearance. Often, in oceanic regions, pilots will deviate within course tolerance (±5-10nm) for weather avoidance without coordinating with ATC (number of deviations, magnitude of deviations, NIEC).

4. Cockpit vidéo (en route phase, NIEC). 5. Instances of and techniques used for cockpit resource management and crew coordination (NCAR

observers). 6. Pilot background, experience (NCAR, questionnaire).

A notional one-hour de-briefing period will follow each flight demonstration. Any further discussion or pilot feedback will be documented for the final report.

45

Testing/Demonstration Areas

Results from the simulated Line-oriented Flight Training (LOFT) will begin the concept of use baseline, providing the FAA and NCAR with insights into:

1. Current oceanic flight operations with pre-flight pilot information briefings (PIBs) and available weather updates;

2. Limitations to decision support for pilots, dispatchers, and air traffic controllers/managers posed by the current operational environment;

3. Flight safety issues related to weather hazards related to flight safety issues; 4. Potential changes to the operational procedures when in-flight updates of weather are available on

a contract, demand, or broadcast service; 5. Potential impacts to ATM and safety. 6. Refined objectives for a future operational oceanic flight demonstration.

Testing/Demonstration Procedure Format

The LOFT simulator missions will approximate as much as possible an actual oceanic airline flight. Repositioning of the aircraft will be kept to a minimum so that pilot task flow remains realistic. Demonstration objectives will be addressed mostly through observations by the evaluators, plus having the pilots share their opinions of the value of weather information, display presentation, or reasons for selecting a particular course of action. Detailed discussions of aspects of the demonstration will be done post-mission as much as possible.

46

Flight One—Baseline

Each evaluation mission will be preceded by a 1-hour pre-mission briefing. This time will be used to review evaluation procedures and pilot expectations from weather information, ATC, and airline dispatch. Pilots will also be trained on weather display graphics, their meaning, and display capabilities. During the flight scenario, normal flight procedures will be used and observed through climb-out and level-off. The flight scenario will include the following procedures and normal ATC/dispatch communications:.

1. Clearance delivery, crew briefing, engine start, pre-taxi.. 2. Before take-off checklist (note: the flights began at lineup, on the departure runway, due to

limitations encountered with the out-the-window visual display). 3. Take-off clearance, lineup checklist, normal take-off. 4. After take-off/climb procedures and checklist. Once stabilized on climb-out, the climb to cruise

altitude may be accelerated to en route cruise altitude and airspeed, if time requires. NOTE: Normal procedures may be abbreviated somewhat after more experience with the scenario is obtained during the dry runs.

Once the aircraft is established en route, the notional sequence of events for the baseline flights will be:

1. The first weather scenario will be presented on the airborne weather radar, approximately 120-150nm prior to the leading edge of the event (consistent with the radar look-ahead capability).

2. Pilot response—request for weather updates via text or verbal communication, pilot reference to PIB information, pilot request for reroute or altitude change.

3. Presentation of second weather scenario (which may be a continuation of the first scenario). 4. Pilot response. 5. Resumption of en route procedures through arrival.

• Observations • Reference to PIB information • Pilot requests for information from dispatch and ATC—what and when? • Pilot requests for reroute or deviation. When/how early does the appearance of significant

weather (on the EFB and/or airborne radar) result in a perceived need to deviate? How many deviations are required?

• What was the flight’s closest approach to the weather hazard? • What was the cost in terms of flight time (fuel) resulting from the deviation strategy? • What was the workload impact caused by the weather scenario? One measure of this might be

the number of attempts to request information (either via HF voice or ACARS data communications) from dispatch and/or ATC, or reference to on-board PIB information.

• Communication with flight attendants/passengers; use of seat belt sign.

• Questions • Subjective assessment of workload (pilot, controller, dispatch) impact. • Subjective assessment of safety impact. • What other information would the pilot seek in the baseline case? • Were there any pilot techniques used to supplement the information available to them? • What more information would have helped the decision-making process in the baseline case?

47

Script for Flight One – Ft. Lauderdale to Lima, Peru

Event/Cue ATC Script/Text AOC Script/Text Weather—Radar

Weather—EFB

Pilot Action

KFLL N26044 W080092 Pre-start checklist - checklist - clearance delivery (ATC) - dispatch clearance - FMC setup for departure/en route - weather check - takeoff data - departure briefing

Clnc Del 128.4: Spirit 977 cleared to Lima, Beech 2 departure, BAHMA transition, flight planned route. Climb and maintain FL340, squak 2136, contact departure on 122.5.

Dispatch message—clearance for departure and flight information

Clearance delivery, Spirit 977, request clearance to Lima, we have information Charlie. Read back clearance

Pre-taxi - checklist - taxi clearance

Gnd Control 121.4: Spirit 977 cleared to taxi, runway 9L. Winds 090 at 10. Contact tower on 119.3.

Ground Spirit 977 taxi

Taxi - checklist - crew briefing for departure - Takeoff clearance

Tower 119.3: Spirit 977 cleared for takeoff, winds 090 at 10, contact departure when airborne.

Acknowledge “out” message

Dispatch, Spirit 977 “out” Switch to Tower, 119.3 Tower, Spirit 977, ready for departure.

Lineup, takeoff - switch to departure control - gear/flaps - checklist - establish

Spirit 977, departure, radar contact climb and

Switch to Departure 122.5 Departure, Spirit 977 airborne, climbing to FL360.

48

climb profile - dispatch “off” time report

maintain FL360, proceed direct URSUS

Acknowledge “off” time

Sprit 977 Direct URSUS Dispatch, Spirit 977 off at [time], estimating Lima at [time]

Direct URSUS N24000 W079042

Departure: Spirit 977 Contact Miami Center on 128.225 so long Miami Center, roger

Miami on 128.225, Spirit 977 Switch to Miami 128.225 Miami Center Spirit 977 with you, climbing to FL360 direct URSUS Spirit 977 roger

TOC N24076 W079044 Level-off, en route procedures - checklist

Miami Center, roger, radar contact lost, report URSUS to Havana Center 123.7

Miami Center Spirit 977 level at FL360

URSUS N24000 W079042

Havana Center: Roger Spirit 977 report GAXER Respond to requests for deviation

Respond to requests for weather information.

Wx Scenario 1

Switch to 123.7 Position Report: Havana Center Spirit 977, URSUS at [time], FL360, estimate GAXER at [time] DAGUD next

UL780 PIGBO N21188 W079078

Spirit 977 descend and maintain FL350 report GAXER to Havana Center on 120.25

Descend to FL350, acknowledge to Havana Center

UL780 GAXER N20000 W081320

Switch to 120.25. Position Report: Havana Center Spirit 977, GAXER at [time], FL350,

49

Spirit 977, roger. Report DAGUD to Panama Oceanic on 127.5

estimate DAGUD at [time], BUXOS next

UL780 DAGUD N15000 W079197

Spirit 977 roger, report BUXOS to Panama Radio on 123.6

Continuation of Wx Scenario 1

Switch to 127.5. Position Report: Panama Spirit 977, DAGUD at [time], FL350, estimate BUXOS at [time], UGUPI next

UL780 BUXOS N05101 W079400

Respond to requests for deviation Spirit 977 roger, report UGUPI to Guayaquil on 119.9

Respond to requests for weather information

Follow-on with Wx Scenario 2

Switch to 123.6. Position Report: Panama Spirit 977, BUXOS at [time], FL350, estimate UGUPI at [time], GYV (Guayaquil) next

UL780 UGUPI N01250 W079500

Spirit 977 roger, report GYV to Guayaquil this frequency

Switch to 119.9. Position Report: Guayaquil Spirit 977, UGUPI at [time], FL350, estimate GYV (Guayaquil) at [time], VAKUD next

UL780 GYV S02077 W079520

Spirit 977 roger, report VAKUD to Lima Control on 128.1

Position Report: Guayaquil Spirit 977, GYV at [time], FL350, estimating VAKUD at [time], TRU (Trujillo) next

UL780 VAKUD S04302 W079340

Spirit 977, roger. Climb and maintain FL370. Report TRU.

Switch to 128.1 Position Report: Lima Spirit 977, VAKUD at [time], FL350, estimating TRU at [time], BTE next Spirit 977, roger, leaving FL350 for FL370.

UL780 TRU S08052 W079067

Position Report: Lima Spirit 977, TRU at [time],

50

Flight Two--Enhanced

Each evaluation mission will be preceded by a 1-hour pre-mission briefing. This time will be used to review evaluation procedures and pilot expectations from weather information, ATC, and airline dispatch. Pilots will also be trained on weather display graphics, their meaning, and display capabilities. During the flight scenario, normal flight procedures will be used and observed through climb-out and level-off. The flight scenario will include the following procedures and normal ATC/dispatch communications:.

1. Clearance delivery, crew briefing, engine start, pre-taxi. 2. Before take-off checklist (note: the flights began at lineup, on the departure runway, due to

limitations encountered with the out-the-window visual display). 3. Take-off clearance, lineup checklist, normal take-off. 4. After take-off/climb procedures and checklist. Once stabilized on climb-out, the climb to cruise

altitude may be accelerated to en route cruise altitude and airspeed, if time requires. NOTE: Normal procedures may be abbreviated somewhat after more experience with the scenario is obtained during the dry runs.

Once the aircraft is established en route, the notional sequence of events for the enhanced flights will be:

1. The first weather scenario is presented in the form of a forced uplink, a minimum of 400nm from the leading edge of the weather event. The uplink is in response to a pilot level-off position report and estimated time of arrival for the next waypoint. The weather scenario is presented in both character graphic and color graphic formats on the EFB.

2. Pilot response—request for weather updates via text or verbal, pilot reference to PIB information, pilot request for reroute or altitude change.

3. Corroborated weather information is presented on the airborne weather radar display.

Spirit 977 roger, radar contact.

FL370, estimating BTE at [time], ATOGO next

UG436 BTE S09089 W078313

Dispatch arrival information

UG436 ATAGO S10118 W078006

UG436 TOD S10357 W077489

UG436 GALGO S11108 W077317

LIM S12005 W077074

51

4. Additional pilot response to corroborated evidence of the weather hazard. 5. Presentation of second weather scenario (which may be a continuation of the first scenario). 6. Pilot response. 7. Resumption of en route procedures through arrival.

• Observations

• Reference to PIB information • Pilot requests for information from dispatch and ATC—what and when? • Pilot requests for reroute or deviation. When/how early does the appearance of significant

weather (on the EFB and/or airborne radar) result in a perceived need to deviate? How many deviations are required?

• What was the flight’s closest approach to the weather hazard? • What was the cost in terms of flight time (fuel) resulting from the deviation strategy?

• What was the workload impact caused by the weather scenario? One measure of this might be the number of attempts to request information from dispatch and/or ATC, or reference to on-board PIB information.

• Communication with flight attendants/passengers; use of seat belt sign.

• Questions • Subjective, comparative assessment of workload impact relative to no update. • Subjective, comparative assessment of safety impact relative to no update. • What other information would the pilot seek beyond the weather update? • What more information on the display would have helped the decision-making process? • Subjective contrast (understanding, information transfer, usability) between the character

and color graphic. • Would more altitude contours enhance decision-making? • What additional pilot interaction, display functionality would enhance information

transfer?

52

Script for Flight Two – Ft. Lauderdale to Lima, Peru

Event/Cue ATC Script/Text AOC Script/Text Weather—Radar

Weather—EFB

Pilot Action

KFLL N26044 W080092 Pre-start checklist - checklist - clearance delivery (ATC) - dispatch clearance - FMC setup for departure/en route - weather check - takeoff data - departure briefing

Clnc Del 128.4: Spirit 977 cleared to Lima, Beech 2 departure, BAHMA transition, flight planned route. Climb and maintain FL340, squak 2136, contact departure on 122.5.

Dispatch message—clearance for departure and flight information

Clearance delivery, Spirit 977, request clearance to Lima, we have information Charlie. Read back clearance

Engine start Pre-taxi - checklist - taxi clearance

Gnd Control 121.4: Spirit 977 cleared to taxi, runway 9L. Winds 090 at 10. Contact tower on 119.3.

Ground Spirit 977 taxi

Taxi - checklist - crew briefing for departure - Takeoff clearance

Tower 119.3: Spirit 977 cleared for takeoff, winds 090 at 10, contact departure when airborne.

Acknowledge “out” message

Dispatch, Spirit 977 “out” Switch to Tower, 119.3 Tower, Spirit 977, ready for departure.

Lineup, takeoff - switch to departure control - gear/flaps - checklist - establish climb profile - dispatch “off” time report

Spirit 977, departure, radar contact climb and maintain FL360, proceed direct URSUS

Switch to Departure 122.5 Departure, Spirit 977 airborne, climbing to FL360. Sprit 977 Direct URSUS

53

Acknowledge “off” time

Dispatch, Spirit 977 off at [time], estimating Lima at [time]

Direct URSUS N24000 W079042

Departure: Spirit 977 Contact Miami Center on 128.225 so long Miami Center, roger

Miami on 128.225, Spirit 977 Switch to Miami 128.225 Miami Center Spirit 977 with you, climbing to FL360 direct URSUS Spirit 977 roger

TOC N24076 W079044 Level-off, en route procedures - checklist

Miami Center, roger, radar contact lost, report URSUS to Havana Center 123.7

Miami Center Spirit 977 level at FL360

URSUS N24000 W079042

Havana Center: Roger Spirit 977 report GAXER Respond to requests for deviation

Respond to requests for weather information.

Wx Scenario 1 Wx Scenario 1 Switch to 123.7 Position Report: Havana Center Spirit 977, URSUS at [time], FL360, estimate GAXER at [time] DAGUD next

UL780 PIGBO N21188 W079078

Spirit 977 descend and maintain FL350 report GAXER to Havana Center on 120.25

Descend to FL350, acknowledge to Havana Center

UL780 GAXER N20000 W081320

Spirit 977, roger.

Switch to 120.25. Position Report: Havana Center Spirit 977, GAXER at [time], FL350, estimate DAGUD at [time], BUXOS

54

Report DAGUD to Panama Oceanic on 127.5

next

UL780 DAGUD N15000 W079197

Spirit 977 roger, report BUXOS to Panama Radio on 123.6

Continuation of Wx Scenario 1

Continuation of Wx Scenario 1

Switch to 127.5. Position Report: Panama Spirit 977, DAGUD at [time], FL350, estimate BUXOS at [time], UGUPI next

UL780 BUXOS N05101 W079400

Respond to requests for deviation Spirit 977 roger, report UGUPI to Guayaquil on 119.9

Respond to requests for weather information

Follow-on with Wx Scenario 2

Follow-on with Wx Scenario 2

Switch to 123.6. Position Report: Panama Spirit 977, BUXOS at [time], FL350, estimate UGUPI at [time], GYV (Guayaquil) next

UL780 UGUPI N01250 W079500

Spirit 977 roger, report GYV to Guayaquil this frequency

Switch to 119.9. Position Report: Guayaquil Spirit 977, UGUPI at [time], FL350, estimate GYV (Guayaquil) at [time], VAKUD next

UL780 GYV S02077 W079520

Spirit 977 roger, report VAKUD to Lima Control on 128.1

Position Report: Guayaquil Spirit 977, GYV at [time], FL350, estimating VAKUD at [time], TRU (Trujillo) next

UL780 VAKUD S04302 W079340

Spirit 977, roger. Climb and maintain FL370. Report TRU.

Switch to 128.1 Position Report: Lima Spirit 977, VAKUD at [time], FL350, estimating TRU at [time], BTE next Spirit 977, roger, leaving FL350 for FL370.

UL780 TRU S08052

Position Report: Lima Spirit 977,

55

Post Mission De-brief

At the conclusion of each simulator session, the NCAR observers will conduct a post-mission debriefing. The debriefing time (nominally, one hour) will be spent clarifying pilot questions and NCAR observations during the session. The pilots will be given the opportunity to fill out a questionnaire that will focus on the demonstration objectives and the RCS’ utility as a demonstration or evaluation tool for HOTL methodology development.

W079067 Spirit 977 roger, radar contact.

TRU at [time], FL370, estimating BTE at [time], ATOGO next

UG436 BTE S09089 W078313

Dispatch arrival information

UG436 ATAGO S10118 W078006

UG436 TOD S10357 W077489

UG436 GALGO S11108 W077317

LIM S12005 W077074

56

Appendix D – Post-evaluation Questionnaire and Observer Rating Form

In General:

How well does enhanced weather depiction enable these situational awareness and decision making attributes in comparison to the current system of hardware and procedures?

Monitoring weather along your route of flight?

Making reroute decisions?

Obtaining relevant, timely weather information?

En route weather communication with Dispatch?

En route weather communication with ATC (alter-hdg or reroute)

Safety of flight during adverse weather?

Selecting a divert or deviation course?

General situational awareness?

CURRENT is much more effective

No Difference

Enhanced is much more effective

How many times did you use the system to make inflight decisions?

How many times did you actually deviate (change heading) for weather?

What did you think about the information transfer effectiveness of the character graphic method of display?

Do you foresee any safety of flight issues with conducting concept validation of this capability on actual oceanic trips? Please consider the ASCII display and the color graphic.

57

Would a color graphic that resembles the airborne weather radar display be more effective?

What other weather hazards would be useful if presented in the character graphic format? the color graphic format? (For example: turbulence, icing, wind, volcanic ash, others.)

Can you speak to AOC and ATC information and display needs?

As discussion items, how did the presence of enhanced convection information affect your

Decision-making?

Situation awareness?

Flight safety?

Flight efficiency?

Specifically addressing the frequent temporal update capability, how does this aspect impact the above discussion items?

Decision-making?

Situation awareness?

Safety?

Flight efficiency?

58

Appendix E – Pilot Training Handout

Introduction

The Weather Technology in the Cockpit (WTIC) Transoceanic Human-Over-the-Loop (HOTL) Demonstration will be conducted at the FAA Next Generation Integration and Evaluation Capability (NIEC) Research Cockpit Simulator (RCS) William J. Hughes Technical Center (WJHTC) Atlantic City, NJ.

Purposes of this Demonstration

The purposes of the Transoceanic HOTL Demonstration using the FAA NIEC RCS include

• Evaluating the risk of in-flight evaluations of updated weather information in oceanic/remote regions,

• Increasing our understanding of impacts to pilot, dispatch, and air traffic management (ATM) decision making in a collaborative environment when updated oceanic weather information is provided to the flight deck, and.

• Identifying demonstration objectives that are best accomplished with an expanded demonstration of uplinked hazardous weather information to transoceanic airline flights.

Background

Prior to 2006, the FAA Aviation Weather Research Program (AWRP) sponsored the Oceanic Weather Product Development Team (OW PDT) which was charged to develop aviation weather products specifically designed to meet the needs of transoceanic aircraft. The OW PDT collaborated with United Airlines to successfully demonstrate the usefulness of an uplinked, satellite-based product that identified the 30kft and 40kft convective cloud top heights in a two-hour look-ahead display focused on the aircraft position and flight direction (like Figure 1). An ASCII character display was sent to the Boeing 777 aircraft onboard ACARS line printer when a significant amount of deep convection existed along the flight route. Similarly, the AWRP Turbulence PDT has demonstrated the uplink of a look-ahead turbulence severity product into the cockpit of selected CONUS United Airlines flights.

The recent Air France Flight 447 accident has focused attention to the need for additional, aircraft-specific weather information in the cockpit, particularly for transoceanic flights.

Figure 1 shows a reconstruction of the cloud top height product that could have been uplinked at the current position waypoint (shown as a red “X” in the bottom center of each panel), prior to the aircraft encounter with the mesoscale convective complex.

59

Figure 1. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 1 June 2009 at 0115 UTC via an ASCII, line printer graphic (left) and a color-coded graphic (right) relative to the last known position of Air France Flight 447 (bottom center). The 30Kft contour is represented by a “/” and green shading; the 40Kft contour by a “C” and red shading The images are drawn relative to the expected flight route over the next 2 hours.

Prior to uplink of actual weather to en route aircraft, and to support development of a Concept of Use for uplinked oceanic weather, a focused ground demonstration of this capability using line airline pilots would be beneficial. The integration of “canned” weather information into pilots’ decision processes in the ground simulator environment is planned with the FAA NIEC RCS. This ground evaluation will better focus the in-flight phase on the most relevant objectives identified during test planning.

Objectives

• Train flight crews on the capabilities and limitations of planned uplink weather and representations presented on the flight deck. Identify content needed in the flight demonstration Flight Crew Bulletin (FCB) and crew training. Uplinked weather information in oceanic/remote areas may be limited in terms of information content, spatial and temporal resolution when compared to that provided in less data-sparse environments in continental regions.

• Identify those decisions pilots make in the current environment without weather updates, and propose decisions that can be facilitated with frequent weather hazard updates while en route in oceanic/remote regions. Specifically

60

o How does updated weather information affect timing of deviation request from the pilot?

o How does the updated information enhance operational safety? That is, does the availability of updated weather hazard information (in addition to that provided by the airborne radar) decrease the flight’s exposure (as measured by radar range) to hazardous weather during an encounter?

o Does a timely update result in a reduction of flight time and/or fuel burn? o Does the passive uplink of weather updates reduce pilot communications with

dispatch and with air traffic control? • Obtain initial flight crew feedback on weather hazard needs and display presentation

concepts. Specifically, o Is the ACARS text graphic adequate for conveying basic hazard information? o Does the increased potential for information transfer offered by a graphic

electronic flight bag (EFB) display provide additional efficiency and safety benefit?

• Identify situations where collaborative decisions between air traffic controllers, dispatch, and pilots using common, updated oceanic weather hazard information can benefit operations in oceanic/remote regions.

• Build upon demonstration objectives that can be addressed from the uplink of updated weather hazard information (specifically, turbulence and convection) to operational airline flights in oceanic/remote regions.

Flight One—Baseline

Each evaluation mission will be preceded by a 1-hour pre-mission briefing. This time will be used to review evaluation procedures and pilot expectations from weather information, ATC, and airline dispatch. During the flight scenario, normal flight procedures will be used and observed through climb-out and level-off. The flight scenario will include the following procedures and normal ATC/dispatch communications:

• Pre-start checklist. • Clearance delivery, engine start, pre-taxi, crew briefing. • Before take-off checklist (note: the flights began at lineup, on the departure runway, due

to limitations encountered with the out-the-window visual display). • Take-off clearance, lineup checklist, normal take-off. • After take-off/climb procedures and checklist. Once stabilized on climb-out, the climb to

cruise altitude may be accelerated to en route cruise altitude and airspeed, if time requires.

61

Once the aircraft is established en route, the notional sequence of events for the baseline flights will be:

• The first weather scenario will be presented on the airborne weather radar, approximately 120-150nm prior to the leading edge of the event (consistent with the radar look-ahead capability).

• Pilot response—request for weather updates via text or verbal, pilot reference to PIB information, pilot request for reroute or altitude change.

• Presentation of second weather scenario (which may be a continuation of the first scenario).

• Pilot response. • Resumption of en route procedures through arrival.

• Observations o Reference to PIB information o Pilot requests for information from dispatch and ATC—what and when? o Pilot requests for reroute or deviation. When/how early does the stimulus result in

a perceived need to deviate? How many deviations are required? o What was the flight’s closest approach to the weather hazard? o What was the cost in terms of flight time (fuel) resulting from the deviation

strategy? o What was the workload impact caused by the weather scenario? One measure of

this might be the number of attempts to request information (either via HF voice or ACARS data communications) from dispatch and/or ATC, or reference to on-board PIB information.

• Questions o Subjective assessment of workload impact. o Subjective assessment of safety impact. o What other information would the pilot seek in the baseline case? o What more information would have helped the decision-making process in the

baseline case? Flight Two--Enhanced

Each evaluation mission will be preceded by a 1-hour pre-mission briefing. This time will be used to review evaluation procedures and pilot expectations from weather information, ATC, and airline dispatch. During the flight scenario, normal flight procedures will be used and observed through climb-out and level-off. The flight scenario will include the following procedures and normal ATC/dispatch communications:

• Pre-start checklist. • Clearance delivery, engine start, pre-taxi, crew briefing. • Before take-off checklist (note: the flights began at lineup, on the departure runway, due

to limitations encountered with the out-the-window visual display). • Take-off clearance, lineup checklist, normal take-off.

62

• After take-off/climb procedures and checklist. Once stabilized on climb-out, the climb to cruise altitude may be accelerated to en route cruise altitude and airspeed, if time requires.

Once the aircraft is established en route, the notional sequence of events for the enhanced flights will be (Figure 2):

• The first weather scenario is presented in the form of a forced uplink, a minimum of 400nm from the leading edge of the weather event. The uplink is in response to a pilot level-off position report and estimated time of arrival for the next waypoint. In a FANS environment, this report would be automatic. The weather scenario is presented in both character graphic and color graphic formats on the EFB.

• Pilot response—request for weather updates via text or verbal, pilot reference to PIB information, pilot request for reroute or altitude change.

• Corroborated weather information is presented on the airborne weather radar display. • Additional pilot response to corroborated evidence of the weather hazard. • Presentation of second weather scenario (which may be a continuation of the first

scenario). • Pilot response. • Resumption of en route procedures through arrival.

• Observations o Reference to PIB information o Pilot requests for information from dispatch and ATC—what and when? o Pilot requests for reroute or deviation. When/how early does the stimulus result in

a perceived need to deviate? How many deviations are required? o What was the flight’s closest approach to the weather hazard? o What was the cost in terms of flight time (fuel) resulting from the deviation

strategy? o What was the workload impact caused by the weather scenario? One measure of

this might be the number of attempts to request information from dispatch and/or ATC, or reference to on-board PIB information.

• Questions o Subjective, comparative assessment of workload impact relative to no update. o Subjective, comparative assessment of safety impact relative to no update. o What other information would the pilot seek beyond the weather update? o What more information on the EFB would have helped the decision-making

process? o Subjective contrast (understanding, information transfer, usability) between the

character and color graphic. o Would more altitude contours enhance decision-making? o What additional pilot interaction, display functionality would enhance

information transfer?

63

Appendix F – Weather Scenarios for 2 July, 27 August, 2010

Appendix F is included to further illustrate the pilot responses to the weather cases, which were analyzed in Section 3. Figure 8 and Figure 9 are repeated to lead off the waypoint-to-waypoint illustration for each weather case. Two strategies emerge from the more detailed analysis—first, the enhanced safety resulting from avoiding deep convection by making strategic deviations earlier; and second, reducing the resulting deviation from flight plan. These two strategies seem to be a tradeoff. Finally, pilots became more adept at using the CTOP as a strategic addition to the airborne radar information (verses a stand-alone avoidance tool) as each flight progressed (see Section 3). In each case, the CTOP graphic is reoriented approximately top-down to coincide with the CTOP weather scenario.

Figure F-1. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 2 July 2010. Crew 1 (baseline) and Crew 2 (uplink) flight tracks for the entire route are shown.

64

65

66

67

68

69

70

71

72

73

Figure F-11. Graphical depiction of the GOES-East derived cloud top heights (30Kft and 40Kft contours) from 27 Aug 2010. Crew 1 (uplink) and Crew 2 (baseline) flight tracks for the entire route are shown.

74

75

76

77

78

79

80

81

82

83

84