A practical tool to enhance the effectiveness of debriefing 1

A practical tool to facilitate the performance analysis of sharp-end operators and enhance the effectiveness of debriefing.

Stefano Cignoni

Author Note

Capt. Stefano Cignoni; A320 TRI / TRE; CRMIE; Human factors specialist.

Correspondence concerning this article should be addressed to Stefano Cignoni, e-mail: scignoni@alice.it .

A practical tool to enhance the effectiveness of debriefing 2 Abstract

Trainers and facilitators often lack the ability to get the most from the learning opportunities that a debriefing session offers. This paper outlines an alternative way to manage a debriefing session with the use of a graphical tool based on the threat and errors management framework. This tool helps trainers/facilitators focus on the facts in the context reconstruction process thus enhancing the effectiveness of debriefing. (Woods, Dekker, Cook, Johannesen, Sarter, 2010). It also encourages and facilitates trainees/operators to review their own performance so that they can maximize learning and increase their resilience and skills in handling new operational situations. (Weick, Sutcliffe, 2007) Several training and actual operation debriefing sessions were addressed using this tool which turned out to be useful in terms of reconstruction of events and performance analysis.

Key words: Threat and Errors management; debriefing; training; learning; performance analysis; crew resource management; behaviors; resilience; notechs; evidence based training.

A practical tool to enhance the effectiveness of debriefing 3 Introduction

Debriefing sessions in training and real world operations (aviation, surgery, marine, military) is crucial for trainee/sharp-end operators to maximize learning, review and assess their own performance and widen their expertise. It is also important in assessing the organizational culture and receiving feedback on the operational and training policies of the organization itself. (Reason, 1997; ICAO, 2013) Practical tools used in a debriefing session, such as the one proposed and described in this paper, can greatly contribute to identify and learn the essential behaviors (technical and non-technical) needed for effective performance leading to safe operations. It could be also beneficial in increasing operator's resilience abilities in handling operational complexity and unexpected events. (Dekker, Hollnagel, Woods, Cook, 2008; Henley, 2003)

This paper is based on Flight Crew as front line/sharp-end operators. The words "trainees/operators" are used in a general sense to refer to them both in training and in operational context.

Debriefing sessions should involve a fair and unbiased performance review based on observed behaviors and facts. Instruction and facilitation techniques, when properly used, are the most effective way to manage this phase. (Dismukes, Smith, 2000)

The use of questions, cameras, recorded video, interaction management, manuals, written procedures and interviewing techniques are all effective ways to handle a debriefing sessions and to review the trainee/operator’s performance. (Dismukes, Smith, 2000)

The idea of further improving this stage with a common pattern to be used as a shared guide comes from the operational experience. The attention has been addressed to some typical issues of this moment: subjectivity, the inability to build a fair and non-threatening environment in which both trainees/operators and instructors/facilitators are free to discuss and review past performances with a constructive approach, talk about feelings and impressions rather than facts and behaviors, training effectiveness, etc..

A practical tool to enhance the effectiveness of debriefing 4 To fix these issues a graphical tool based on Threat and Error management framework (TEM)

has been developed. The aim is to help trainees/operators, and instructors/facilitators, to thoroughly reconstruct the context in which the performance was observed, identify the behaviors needed to handle the operational complexity, increase learning opportunities and develop task strategies and team interaction abilities that will allow them to share the same accurate mental model of the “true state of the world".(Smith-Jentsch, Cannon-Bowers, Tannenbaum, Salas, 2008)

“TEM can be described as an overarching safety concept regarding aviation operations. The three basic components of the TEM framework are:

‘Threats’ or “events or errors that occur beyond the influence of the flight crew, increase operational complexity, and which must be managed to maintain the margins of safety”.

‘Errors’ or “actions or inactions by the flight crew that lead to deviations from organizational or flight crew intentions or expectations”.

‘Undesired (aircraft) states’ or as “flight crew-induced aircraft position or speed deviations, misapplication of flight controls, or incorrect system configurations, associated with a reduction in margins of safety”. (Maurino, 2005)

The TEM framework “proposes that threats and errors are part of the everyday aviation operations that must be managed by flight crews, since both threats and errors, have the potential to generate undesired (aircraft) states. Undesired aircraft state management represents the last opportunity to prevent an unsafe outcome or end state”. (Maurino, 2005)

Crew Resource Management is the effective use of available resources, where emphasis is placed on the non-technical aspects of crew performance. In relation to TEM, CRM behaviors are an effective tool for managing threats and errors. TEM can thus be understood as encompassing CRM with respect to crew performance or, in other words, CRM can be seen as being focused on TEM. (EASA 2014)

A practical tool to enhance the effectiveness of debriefing 5 The use of this tool has two main phases: the first one is the bottom-up “context's

reconstruction” phase, and the second one is the “management analysis” top-down phase.

In the first one the instructor/facilitator together with trainees/operators will objectively identify and describe the outcome (if any), the undesired (aircraft) state, the behaviors (action, inaction, interaction) that contribute generating the undesired (aircraft) state, and the threats in the context that increased the operational complexity. It is important to remember that in a given context, trainees and operators behave according to the local rationality principle depending on their knowledge, mindset, and different conflicting goals. Behaviors (action, inaction, interaction) are considered erroneous, in other words classified as an error, only in hindsight. Before this, the trainees/operators’ behaviors (action, inaction, interaction) are consistent with their perspective, knowledge and their focus of attention, and make sense to them at that time and in that context without the intention of making a mistake. (Woods, Dekker, Cook, Johannesen, Sarter, 2010) That’s why is utmost important to address this phase properly. Because in doing so the operator perspective is clearly described, identified, and shared among the participants using the tool as a common guide to collect all the factual items. (Dekker, 2006)

In the second top-down “management analysis” phase, the target is to help trainees/operators to identify the behaviors that would have been useful in that context to manage the operational complexity. To make this task easier the use of the NOTECHS model behaviors list (Flin, O’Connor, Crichton, 2008) or EBT Core Competencies and Behavioral Indicators (ICAO Doc.9995, 2013) could be helpful in recognizing and sharing the behaviors needed.

The objective is to thoroughly discuss each "management" phase and consider them as the most important moment for training/reviewing purposes. These phases represent the core and purpose of the debriefing session itself. It is in discussing these phases that the competence of the trainees/operators effectively grows and expands. (Helmreich, Klinect, Wilhelm, 1999; Thomas, 2004)

A practical tool to enhance the effectiveness of debriefing 6 Trainees/operators must be aware of the human limitations as well as of the possibility of

making mistakes. (Reason, 1990) If they learn to review their performance using this tool, they will also be able to use, as an habit, the same schema and behaviors, even in real life operations. In other words look for possible threats that could increase the operational complexity (threats seeking), evaluating them in the right way in terms of risk assessment, manage them accordingly, (Anticipate) (Weick, Sutcliffe, 2007; Klein, 1999; Endsley, 2000) manage the potential errors (in response to a mismanaged threat, or a spontaneous one) and manage the possible consequential undesired states. (Contain) (Weick, Sutcliffe, 2007; Klein, 1999; Endsley, 2000).

Crew Resource Management is the bedrock of Threats and errors management. It represents, in terms of knowledge and behaviors, the resources needed by trainees/operators to address the operational complexity . The link among CRM, TEM and the proposed tool is represented by the BEHAVIORS boxes on the left side of every management phases. In that boxes should be listed all the behaviors used, or needed, to manage the relative phases in order to make threats, errors, and undesired states inconsequential. (Countermeasures)

This tool also reflects the debriefing session management best practices. The traditional model of debriefing analysis "What happened? How it happened? Why it happened? What can we learn?" can be found in the Context reconstruction as well as in the Management analysis phases.

The first two questions (What happened? How it happened?) are differently addressed in the Context reconstruction phase in order to facilitate the process of collecting facts and encouraging interactivity among participants. In particular, “What happened?” is the “outcome” or “undesired state” condition of the Threat and error management model. “How it happened?” instead, refers to the behaviors, (in hindsight errors) either spontaneous, or in response to threats, that trainees/operators were trying to manage.

A practical tool to enhance the effectiveness of debriefing 7 “Why it happened?” and “What can we learn?” are addressed in the Management analysis

phase, in each “management” part of the model (Threats management, errors management, and undesired state management) as the most important learning part of the debriefing session.

It’s important to remember that this way of handling a debriefing session is centered on trainees/operators. In the “management analysis” phases, to maximize learning, instructors and facilitators should avoid giving their opinion, but instead they should lead the trainees/operators to self-discuss and evaluate their own performance by using the proposed tool. (Dismukes, Smith, 2000; Woods,2013)

Basic debriefing techniques need to be well known and mastered by instructors and facilitators. Starting to ask questions to the lower in rank, or junior crew member so as to avoid influences from the senior one, is one of the best practice of debriefing technique. The proper types of questions (open, close, leading, ecc.), listening techniques, effective use of silence, the use of body language, follow up phrases, and so on, used with the proposed tool could greatly improve and enhance the quality of the debriefing session. (Dismukes, Smith, 2000)

The aim of this tool, and this way of managing the debriefing session, is to train trainees, and help operators to review their course of action from their perspective, or point of view, analyzing what they could have done differently in that moment, and in that context to address the undesired states, errors, and threats thus increasing their resilient abilities leading to safe operations. (Woods, 2013)

Tool

The model is graphically depicted as an inverted triangle, so as to represent the sharp end of the System where the trainee/operator directly interacts with the hazardous process (close in space and time). It also represents the trainee/operator’s attention, time, and cognitive resources required to handle the situation at hand as he/she moves toward the lower part of the model. Usually when the operator is managing an undesired state has less attention, less cognitive resources and less time

A practical tool to enhance the effectiveness of debriefing 8 available than in the error or threat management phases. (Endsley, 2000) The blunt end of the system, (far in space and time) above the triangle represents the Organizational environment inherited by the trainee/operator that contributes to shape, in terms of constraint and resources, the way he handles the operational complexity. (Woods, Dekker, Cook, Johannesen, Sarter, 2010) . In other words “the hand of cards that the trainee/operator uses to play his game”. Our focus however, is on the sharp end of the model. (Fig.1)

Fig.1

How to use

First we start with the bottom-up Context reconstruction phase (Fig.2) by listing and describing all the factual conditions. In this way we define the context in which trainees/operators have operated. The starting point is from the bottom of the tool, from the lowest level that was reached in actual operations. It could be an outcome, or an undesired state, or, if one or both are missing, it

A practical tool to enhance the effectiveness of debriefing 9 could be a behavior (in hindsight, eror). The following questions might be helpful in addressing this phase:

• Was there any outcome? (fact). If YES, which one? (Clear and shared technical consequence)

• Was an undesired state reached? (facts) (Clear and shared technical consequences) • If YES, which type? (Aircraft handling, Ground navigation, incorrect aircraft

configuration). (Clear and shared technical consequences) The next step is to move up to the behaviors level, listing all the behaviors (action, inaction,

interaction) performed by each trainee/operator that have contributed to generate the undesired state. (ICAO 2002) Here a distinction must be made between spontaneous behaviors (not directly linked to a specific threat), and those performed in response to a threat.

• Do you recall which behavior/s (action, inaction, interaction) performed by whom have led to the undesired state?

• When did it happen? • How could you classify the behavior? (procedural, communication, or handling)

The last step is to work at Threats level, and identify/classify all the multiple conflicting threats existing in the evolving context which contributed to increase the operational complexity.

• What were the threats in the context? • Which type of threats were present in the context?(Environmental, organizational,

unexpected, latent, anticipated) • What threats were perceived as more risky? Open questions are directed at both crew members, starting with the junior one in order to

avoid influencing the answers and for creating a climate that will help to facilitate the context reconstruction. There is no need at this stage to enter into discussions or determine who, or what, is right or wrong, it is important to stay focused on facts. (Dismukes, Smith, 2000)

A practical tool to enhance the effectiveness of debriefing 10 Thus in addition to correctly reconstructing the context, we create a favorable climate for the

next phase, since the focus is not on subjective feelings or impressions, but on shared and identifiable factual data and technical consequences. The use of a whiteboard where it is possible to write down all the relevant statements of the trainees/operators together with the drawing of the model, will help to create the right climate. It's essential keep the discussion fair and focused on facts in order to induce participants to self-critique and active listening.

Fig.2 (Context Reconstruction)

Once we have completed the Context Recostruction phase (Fig.2) by collecting all the factual information, we continue with the top-down “Management analysis” phase. (Fig.3)

Each “Management” part of the tool is represented as an inverted trapezium since not all threats, errors, or undesired states are handled in an inconsequential condition, but some of them are mismanaged, thus producing, in sequence, errors, undesired states, or an outcome.

A practical tool to enhance the effectiveness of debriefing 11 Beginning with the threat management level, starting from the junior Crew member, and then

to the crew as a whole, we first ask each participant to list which behaviors would have been useful in that moment to manage the threats and make them inconsequential, thus preventing (avoid) an error (threat management):

• How was the threat managed? • What could you have done at a threat management level to make the threat

inconsequential? • What would have been useful to do to identify and manage the threats? • Which behaviors would have been useful in that moment for an effective threat

management? • Why was the threat mismanaged, or why did it go undetected? Then we go down to the error management level to determine which behaviors would have

been useful to manage (trap) the error - be it spontaneous, or linked to mismanaged threats - and make it inconsequential, or to determine which behaviors led to additional errors.

• How was the error managed? • What could you have done to make the error inconsequential? • Which behaviors would have been useful in that moment for an effective error

management? • Did you detect the error? If not, why not? • Did it become consequential and did it lead to an UAS? • Did you make any additional errors? • Why was the error mismanaged, or why did it go undetected? In the Context Reconstruction phase we have used the term “behaviors” (actions, inaction,

interaction) referring to what has produced the undesired state. Now we refer to these behaviors as “errors”, and of which type (procedural, communication, or handling), because the performed behaviors that made sense to the trainee/operator in the moment they were attempting to manage the

A practical tool to enhance the effectiveness of debriefing 12 operational complexity (threats), are in hindsight incorrect, having failed to produce the expected results. (Woods, Dekker, Cook, Johannesen, Sarter, 2010)

The word "violation", and the type (routine, optimizing, situational, exceptional), will be used instead, if the action, deliberately did not comply with a rule, procedure, or norm. (Reason, 2008; Whittingham, 2004; Dismukes, 2009)).

The number of violations (and the type) resulting from the analysis is an important indicator of the operational health of the System. (Reason, 1997; ICAO 2013; Degani, Wiener, 1994)

Eventually, at the undesired state management level we use the same process: identify which behaviors would have been useful to manage (mitigate) the undesired state caused by a mismanaged error, and make it inconsequential. We must bear in mind that this is the last opportunity for the trainee/operator to avoid an outcome thus acting as the last line of defense.

• What more could you have done to handle the undesired state? • Which behaviors would have been useful in that moment to handle the undesired State? • Did you make any additional errors? • Was there any threat that brought you directly into an undesired state? • Why was the undesired state mismanaged, or why did it go undetected? The three downward arrows represent respectively: mismanaged threats, mismanaged errors,

and mismanaged UAS (Fig. 3).

All the management phases can be handled by referring to the NOTECHS (Flin, O’Connor, Crichton, 2008) or EBT behaviors list (ICAO Doc.9995, 2013) in order to clearly identify and share the behaviors needed to manage threats, errors, and undesired states. The “learning by identification” technique is used in order to help building the proper competence of the trainees/operators. (Flin, O’Connor, Crichton, 2008; Flin, Martin, Goeters, Hörmann, Amalberti, Valot, Nijhuis, 2003; Henley, 2003)

A practical tool to enhance the effectiveness of debriefing 13

Fig.3 (Management Analysis)

Whenever we are dealing with the “management” level (threats, errors, and undesired state), it is essential to ask why the trainee/operator was not able to “manage” that level: was it a matter of knowledge? (lack of knowledge, inert knowledge, knowledge calibration, buggy knowledge, etc.). Was it a mindset issue? Or were there some conflicting goals that the trainee/operator was trying to balance? It is important to understand the evolving situation and relative context from the trainee/operator’s point of view, in accordance with the local rationality principle. We need to find out why trainee/operator made sense of the situation and behave the way he did. (Woods, Dekker, Cook, Johannesen, Sarter, 2010)

In the end we must try to understand why, in that operation, there were one or more downward arrows, and why the trainee/operator was not able to effectively handle one or more management phases.

In a training environment, to make the learning sessions more effective (learning for success training), we should also refer and reinforce those behaviors that made the threats, errors, and

A practical tool to enhance the effectiveness of debriefing 14 undesired states inconsequential (Woods, 2013). In other words we need to positively highlight where the operator worked effectively in the management phases (inconsequential leftwards arrows).

The following is a list of additional general questions (not exhaustive) to be used by the instructor/facilitator to lead the discussion and trainee/operator’s self- reviewing thus making the debriefing/interview sessions more effective and interactive. They should be used in close connection with appropriate facilitation techniques, as those discussed before, in order to delve further into the analysis:

What kind of knowledge would have been useful in order to tackle the situation at hand? What kind of knowledge do you think would have been useful at that time? What kinds of difficulties did you experience in recalling the knowledge you needed? What was your goal at that time? Where was your focus? Who detected the threat/error/undesired state? Who responded to the error/undesired state? Who was aware of the threat/error/undesired state? For each threat which kind of risk assessment has been made ? Was the crew aware of the threat/error/undesired state? Why did the error/undesired state go undetected? What were you focusing on? What did you see? What were you looking at? What kinds of conflicting goals were you trying to balance? What would you have done differently to avoid…. What did you do well? What could have been done better?

A practical tool to enhance the effectiveness of debriefing 15 Why was it effective? Why wasn’t it effective? How did you communicate among yourselves? How will you do it differently next time? Which CRM behavior does this relate to?

The aim is to provide trainees and operators with skills in TEM principles in order to expand their knowledge about behaviors needed to turn any "management" trapezoid into a triangle by closing the lower edge. That is to say being able to manage all the threats, errors or undesired states in context, in an inconsequential condition, thus ultimately preventing an outcome. (Helmreich, Klinect, Wilhelm, 1999)

The ability to turn threats, errors and undesired states into an inconsequential condition gives us an indication about the knowledge of the trainee/operator's behaviors and his/her proficiency in putting them in place, in other words his/her resilient ability. (Woods, 2013)

Would another trainee/operator in the same context, dealing with the same threats, produce the same results? If the answer is yes, an organizational approach would be the best way to solve the issues (training programs, workplace condition, organizational factors, ecc), otherwise a training or retraining solution would be more indicated. (Strauch, 2002,2004; )

This debriefing tool used as a guide turned out to be useful in training operations as well as in several interviews with crews involved in operational events as a way to standardize the reviewing and keep the discussion fair and centered on factual data.

A practical tool to enhance the effectiveness of debriefing 16 Case Study (Simulator LOFT scenario)

Take off from RWY AA with RWY BB take off data inserted in the FMGS at XXX airport.

Narrative

The crew was completing the pre-flight checks for the scheduled flight. There was a tight departure slot . Runway in use was BB (3200mt long). First Officer calculated the performance take off data for runway BB and loaded them in the FMGS. He also calculated the take-off data for runway AA as well (2200mt long, same flap position but different speeds and take-off thrust setting). He used the paper runway table instead of ACARS interface as usual. At XXX airport ACARS interface is outside coverage so it’s impossible to use it to compute take off data. The wind was such as to permit departure from both runways. Taxi distance from the gate to runway BB holding point is longer than taxi distance to runway AA holding point. Different take-off data and the opportunity to depart from both runways was discussed in the pre departure take-off briefing performed at the gate in order to avoid long taxi time. Once the flight had started taxi for runway BB the crew received from Air Traffic Control 20 minutes departure delay due to ILS ground equipment calibration check on runway BB. The crew then requested to depart for runway AA in order to avoid delay due to departure slot (short taxi distance). F/O started to reconfigure the FMGS with the new speeds, flap position, and take-off thrust setting but inadvertently reinserted runway BB take off data. No pre departure briefing update was carried out by the Captain (who was Pilot flying) and all this tasks were carried out while the airplane was in motion. ATC cleared the flight for take-off. The flight entered runway AA and started the take-off run. The flight lifted off about 600mt from runway end. This late lift-off prompted the crew to check the take-off data used for departure. They realized that they used the wrong data.

Now we describe this event using the Tool and the statement of the Crew obtained from the debriefing session.

A practical tool to enhance the effectiveness of debriefing 17 Context reconstruction phase

Using the questions of the context reconstruction phase together with the appropriate facilitation techniques, the instructor/facilitator will guide the Crew in the self-reconstruction process of the context in which the operation had taken place. It will be helpful, as mentioned before, to draw on a whiteboard all the statements of the crew arranged according to the schema reported in Fig.2:

Outcome: take-off from RWY AA with RWY BB take off data inserted in the FMGS.

Undesired Aircraft State: (aircraft configuration) FMGS incorrectly loaded, aircraft not properly configured for RWY AA take off. (Thrust setting, flap/slat configuration, reference speed).

Crew behaviors (action, inaction, interaction): propose runway change (procedural), update FMGS with aircraft in motion (procedural), other runway take off data inserted but not cross checked by the other crew member (procedural, communication), updated take off briefing for the new runway not performed (procedural, communication), aircraft never stopped before take-off (aircraft handling), before take-off check list not performed again after the new take-off data insertion. (procedural)

Threats: perceived time pressure for possible departure delay due to radio equipment check on RWY BB (Environmental, unexpected) and departure slot (Environmental, anticipated). RWY AA take off data properly computed with runway tables, but graphically reported very close to those of RWY BB on the flight plan format (organizational, latent). ACARS interface not working at airport XXX for take-off data computation (organizational, anticipated). Taxi distance very short for RWY AA, very long for RWY BB (environmental, anticipated).

A practical tool to enhance the effectiveness of debriefing 18 Management analysis phase

Having completed the Context reconstruction phase and identified and classified all the factual conditions and technical consequences (outcome, undesired state, behaviors and threats), the instructor/facilitator will continue now with the Management analysis phase starting from threat management. Using the questions listed at Pag. 11, he will guide the self-discussion and self-analysis of the crew’s performance to list the behaviors needed in that situation to manage the threats. The link with CRM behaviors and the threat and errors management construct is reinforced in this phase by comparing the proposed behaviors stated by the Crew, with those specified in the NOTECHS behaviors list (Flin, O’Connor, Crichton, 2008) list or EBT Core Competencies and Behavioral Indicators (ICAO Doc.9995, 2013). The “context reconstruction” phase of this tool will use the TEM framework to identify the practical consequences, the factual items. The “management analysis” phase will help the crew to focus on the behaviors needed to manage what has been identified in the previous one and then compare these behaviors with the NOTECHS behaviors list (Flin, O’Connor, Crichton, 2008) or EBT Core Competencies and Behavioral Indicators (ICAO Doc.9995, 2013).

The aim here is to help the crew to correctly assess the threats in the context in order to evaluate them in the right way, and build up or reinforce the knowledge of CRM behaviors needed to address them.

For example, some issues that the Crew discussed were related to the need to request an extension of the slot in order to manage the perceived time pressure, and to the opportunity to clearly write in two separate sheets of papers the different take-off data and not on the same one to avoid confusion. Another issue was related to the opportunity to load the FMGS with the performance data of the closer runway (AA) so as to have more time available to update them in case of runway change. According to the previous list of identified issues, the Crew was then guided to identify in the NOTECHS behaviors list (Flin, O’Connor, Crichton, 2008), or EBT Core Competencies and

A practical tool to enhance the effectiveness of debriefing 19 Behavioral Indicators (ICAO Doc.9995, 2013), which behaviors could have been applicable in that context to handle the indicated threats. Then we go down to the error management level (in hindsight, error). Similarly as threats management phase discussion, the instructor/facilitator will guide the self-discussion and self-analysis of the crew’s performance in identifying the behaviors needed in that situation to manage the errors (or prevent additional errors), thus making them inconsequential, and comparing the proposed behaviors reported by the Crew with those indicated in the NOTECHS behaviors list (Flin, O’Connor, Crichton, 2008) or EBT Core Competencies and Behavioral Indicators (ICAO Doc.9995, 2013).

At this level, some outlined issues were related to the need to stop the aircraft, to update the FMGS and cross-check it, to update the take-off briefing together, to read again the before take-off check list, ecc. Again the Crew was guided to identify in the NOTECHS behaviors list (Flin, O’Connor, Crichton, 2008), or EBT Core Competencies and Behavioral Indicators (ICAO Doc.9995, 2013), which behaviors could have been applicable in that context to handle the indicated errors.

At last, at the undesired aircraft state management level, with the same process specified in the threats and errors management phases, the instructor/facilitator will guide the Crew in identifying the behaviors needed to handle the undesired state, thus avoiding an outcome, or why, such as in this case, it went undetected.

Summary

Increasing the “human contribution” (Reason, 2008) to Safety by improving the performance of sharp-end operators is a major challenge. Increasing the trainee/operator’s ability to manage any operational situation and not only an inventory of pre-defined situations is one of the ways to obtain that goal. Debriefing sessions are important for trainees/operators to widen their knowledge and skills needed to handle operational variability and review their performance. Helping them to further their knowledge and skills in handling actual demands, their knowledge of CRM behaviors, their awareness and commitment to resilience, is the fundamental target in debriefing sessions. (Dekker,

A practical tool to enhance the effectiveness of debriefing 20 Hollnagel, Woods, Cook, 2008; Woods, 2013) That’s why debriefing techniques need to be well understood and mastered by trainers and facilitators, as well as the use of practical tools such the one proposed in this paper.

“Context reconstruction” is the first and fundamental step in starting the debriefing session using this tool. In this phase the trainees/operators perspective must be properly reconstructed and shared in order to set the stage for the subsequent phase. More thorough will be this phase more effective will be the next one.

“Management analysis” phase, is the core of this tool. When managed according to the technique described in this paper, using the tool, the questions, and the NOTECHS behaviors list (Flin, O’Connor, Crichton, 2008), or EBT Core Competencies and Behavioral Indicators (ICAO Doc.9995, 2013), in the way they are proposed, will allow trainees/operators and instructors/facilitators to thoroughly review past performance, improve competence, learn to effectively manage the operational complexity, and reinforce the strategies necessary to anticipate (threats seeking and management) and contain (errors and undesired state management) the operational challenges. (Weick, Sutcliffe, 2007; Klein, 1999; Endsley, 2000).

The proposed tool, used as described for managing a debriefing session, might be an alternative way to allow sharp-end operators to become the “last line of defense” in the fundamental “human contribution” for safe operations.(Reason, 2008)

A practical tool to enhance the effectiveness of debriefing 21 References

Degani, A., & Wiener, E. L. (1994). Philosophy, policies, procedures, and practices: The Four "P"s of flight deck operations. In N. Johnston, N. McDonald, and R. Fuller (Eds.), Aviation Psychology in Practice (pp. 44-67). Hants, England: Avebury Technical

Dekker, S., Hollnagel, E., Woods D., Cook R. (2008). Resilience Engineering: New directions for measuring and maintaining safety in complex systems Final Report. Lund University School of Aviation.

Dekker, S. (2006). The field guide to understand human error. Ashgate Publishing Ltd

Dismukes, K.R., Smith, G.M. (2000). Facilitation and debriefing in aviation training and operations. Ashgate Publishing Ltd.

Dismukes, K.R. (2009). Human Error in Aviation. Ashgate Publishing Ltd

European Aviation Safety Agency (2014) Notice of Proposed Amendment 2014-17

Endsley, M.R. (2000). Theoretical underpinnings of situational awareness: a critical review. SA Technologies.

Flin, R., O’Connor, P., Crichton, M. (2008). Safety at sharp end. A guide to Non-Technical Skills. Ashgate Publishing Ltd.

Flin, R., Martin, L., Goeters, K.M., Hörmann, H.J., Amalberti, R., Valot, C., Nijhuis, H. (2003) Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors and Aerospace Safety 3(2), 95-117. 2003, Ashgate Publishing

Helmreich, R.L., Klinect, J.R., & Wilhelm, J.A. (1999). Models of threat, error, and CRM in flight operations. In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 677-682). Columbus, OH: The Ohio State University.

A practical tool to enhance the effectiveness of debriefing 22 Henley, M.A. (2003). Aviation education and training. Ashgate Publishing Ltd

ICAO Doc. 9803 Line Operations Safety Audits (LOSA); 1st edition 2002

ICAO Doc. 9859 Safety Management Manual; 3rd edition 2013

ICAO Doc. 9995 Manual of Evidenced-based Training; 1st edition 2013

Klein, G. (1999). Sources of Power: how people make decisions. The MIT Press, Cambridge, Massachusetts, London, England.

Maurino, D. (2005). Threat and Errors management (TEM). Canadian Aviation Safety Seminar (CASS) ; Vancouver, BC, 18-20 April 2005

Reason, J. (1990). Human error. Cambridge University Press. Cambridge. England.

Reason, J. (1997). Managing the risk of organizational accident. Aldershot. Ashgate. England.

Reason, J. (2008). The Human contribution. Ashgate Publishing Ltd.

Smith-Jentsch, K.A., Cannon-Bowers, J.A., Tannenbaum, S.I., Salas, E. (2008) Guided team self-correction: impact on team mental models, processes, and effectiveness. Small group research. Volume 39, Number 3, Sage Publication.

Strauch, B.(2002, 2004). Investigating Human Error: Incidents, Accidents, and Complex Systems. Ashgate Publishing Ltd.

Thomas, M.J.W. (2004). Predictors of Threat and Errors Management: identification of core nontechnical skills and implications for training systems design. The International Journal of Aviation Psychology. 14(2), 207–231

Weick, K.E., Sutcliffe, K.M. (2007). Managing the unexpected. John Wiley & Sons inc.

Woods, D.D. (2013). Building Flight Crew Resilience in CRM Training.

A practical tool to enhance the effectiveness of debriefing 23 Woods, D.D., Dekker, S., Cook, R.I., Johannesen, L., Sarter, N. (2010). Behind Human Error.

Ashgate Publishig Ltd.

Whittingham, R.B. (2004). The Blame Machine: Why Human Error Causes Accident. Elsevier Butterworth-Heinemann. Linacre House, Jordan Hill, Oxford OX2 8DP