team cognition during a sumulated close air support exercise cognition... · 2008. 6. 24. ·...

Defence Research andDevelopment Canada

Recherche et développementpour la défense Canada Canada

Team Cognition During a Simulated Close Air Support Exercise:Results From a New Behavioral Rating Instrument

Jerzy Jarmasz1, Richard Zobarich2, Lora Bruyn-Martin2, andTab Lamoureux2

1Defence Research & Development Canada, Toronto,Canada

2HumanSystems Inc., Guelph, Canada

Defence R&D Canada • R & D pour la défense Canada

Close Air Support (CAS): Overview

US DoD Joint Pub 3-09.3:CAS is air action by fixed- and rotary-wingaircraft against hostile targets that are inclose proximity to friendly forces and thatrequire detailed integration of each airmission with the fire and movement ofthose forces.


Teamwork in CAS

• Distributed

– Coordinated rather thancollective tasks

• Joint

– Always land and air atleast

• Can be multination/coalition

– Common language ≠common procedures

• This is not easy!!


Importance of Combat ID in CAS

• Friendly forces and adversary are close to each other,and close to civilians and infrastructure

• Targets are small, nimble and mobile: individuals,private vehicles, huts, instead of tanks and bunkers

• Coalition missions:

– Allies can be hard to distinguish from adversary

– Pilots and terminal controllers might not havecommon procedures

• Risk of Combat ID failure leading to fratricide orcivilian casualties is very real in CAS (just read thenews!)


Training approach: Distributed MissionTraining (DMT)

• USAF has extensive experience with DMT (a.k.a DMO)

• Canadian Forces Air Warfare Centre standing up its DMTcapability

• Will allow Canadian Forward Air Controllers (FACs) topractice CAS skills more often and with a wider range of(coalition) airframes than currently possible

– Note that FACs tend to be Army personnel in the CF

– FAC is a secondary duty in the CF

• Requirement to assess effectiveness of DMT inimproving team performance in CAS missions


Study: Assessment of Team Performance in CAS

• Goal: develop behaviourally-anchored rating scale(BARS) for team performance in CAS

• DMT event: Exercise Northern Goshawk, August 2007(CAN-US-UK, involving DRDC Toronto, the Air ForceResearch Laboratory & Defence Science & TechnologyLaboratory)

• Focus on Canadian component of event: CAS support ofa Canadian Forces (CF) convoy mission, controlled by aCF Forward Air Controller


BARS instrument development(Jarmasz et al., 2008)

1. Hierarchical Task Analysis of generic CASmissions

• Focus on FAC perspective

• Validated with CF CAS subject matter experts

2. Identification of key team interaction points basedon HTA

3. Development of behavioural ratings for interactionpoints using behavioural markers of teamcognition breakdown (Wilson, Salas, Priest &Andrews, 2007)


Main measurement points

Perform Battle Damage AssessmentCommunicate Options with PilotSituation Updates

5. Post-attack

Transmit CAS brief and communicate remarksTransmit talk-onDeconflict Target AreaDesignate TargetAbort Mission (as needed)

4. Target prosecution

Communicate Options with PilotSituation Updates

3. Non-kinetic CAS support

Receive pilot’s check-in (includes situation brief)2. Pilot Check-in

Understand Situation UpdatesTransmit Immediate CAS Request (as needed)

1. Pre-check in

TasksCAS mission phase


From team cognition to rating scales(adapted from Wilson et al., 2007)

• To what extent were team members working toward the sameends?

• How effective were FAC/others as a team?Cooperation

• How well were team members' knowledge requirementsmanaged?

• How well did team members monitor each other's performance?• How effective was back-up behaviour?• How adaptable were team members to the changing demands of

the situation?

Coordination

• How effective was information exchange?• Was information exchange economical?• Did closed looped communication go as expected?

Communication


Sample BARS: Communication (CAS Brief)

FAC failed to provide sufficient brief & remarks for pilot to complete mission.1

FAC omitted important brief & remarks items that were available to him.2

FAC passed all items of brief & remarks available to him but had to communicate withothers to obtain missing info requested by pilot (e.g. MAXORD, ACA measures)3

FAC passed all items of brief and most key remarks but provided some remarks onlywhen prompted (e.g. danger close initials)4

FAC passed complete and accurate brief (following theatre standard), & provided allkey remarks info (e.g., weapons effects, attack geometry, ACA measures, number ofattempts, level of risk for blue and white forces, danger close initials).

5

Communication: How effective was information exchange?


Sample BARS: Coordination (CAS Brief)

FAC/pilot failed to display a common understanding of the SOPs &brief/remarks.1

FAC/pilot made explicit attempts at coordinating knowledge, and achieved anincomplete common understanding of the SOPs & briefs/remarks.2

FAC/pilot made explicit attempts at coordinating knowledge (e.g., asked eachother questions), and at great effort (e.g., much time spent in discussion) achievedcommon understanding of the SOPs & brief/remarks.

3

FAC/pilot coordinated explicitly and effectively (e.g., discussed coordinatesformats), and achieved a common understanding of the CAS brief SOPs &brief/remarks with little effort.

4

FAC/pilot implicitly coordinated in an effective manner (e.g., did not requirespecial coordination or discussion beyond standard turn-taking), and displayed acommon understanding of CAS brief SOPs and of the brief/remarks.

5

Coordination: How well were team members’ knowledge requirements managed?


Sample BARS: Cooperation (CAS Brief)

FAC/Pilot could not agree on brief SOPs and did not collaborate to ensure pilotreceived & understood brief & remarks; pilot did not have any useable information toproceed with talk-on & attack.

1

FAC/Pilot collaborated poorly to achieve joint understanding of brief & remarks, anddisagreed significantly about brief SOPs; pilot had only a fraction of the requiredinformation for the talk-on and attack.

2

FAC/Pilot collaborated to understand all mandatory CAS brief info, but did notcooperate fully on understanding remarks or had a significant disagreement aboutbrief SOPs; pilot has most but not all info required for talk-on and attack.

3

FAC/Pilot collaborated to understand all relevant CAS brief & remarks info but bothhad slightly different priorities on brief SOPs, which were easily resolved or accepted.4

FAC/Pilot collaborated to ensure CAS brief SOPs (mandatory CAS brief, or thatcollective SA did not require brief) were adhered to and that all required info waspassed & understood for the attack.

5

Cooperation: To what extent were FAC/Pilot working towards the same ends?


Results

• Two raters performed ratings during Ex NorthernGoshawk

• Minor changes on Day 1; data analyzed for Days 2,3, 4 only

• Due to sheer number of ratings, each rater focusedon slightly different ratings subsets

– Thus not all possible ratings collected

• Only qualitative patterns are presented


Summary statistics for subset of ratings

5.00 (0.00)NR5.00 (NA)Cooperation

4.75 (0.50)NR4.67 (0.58)Coordination

5.00 (0.00)NR5.00 (0.00)Communication4.5 Abort mission

NRNRNRCooperation

4.43 (0.55)4.50 (0.58)NRCoordination

4.50 (0.84)4.33 (0.58)4.14 (1.07)Communication4.4 Designate target

4.00 (0.00)4.00 (0.00)5.00 (0.00)Cooperation

4.33 (0.52)4.00 (0.00)4.13 (0.75)Coordination

4.33 (1.12)4.00 (1.00)4.33 (1.00)Communication4.3 Deconflict target area

5.00 (0.00)5.00 (0.00)5.00 (0.00)Cooperation

4.08 (0.90)4.13 (0.83)3.83 (0.83)Coordination

4.29 (0.69)3.83 (1.17)4.22 (0.97)Communication4.2 Transmit talk-on

5.00 (0.00)5.00 (0.00)5.00 (0.00)Cooperation

4.50 (0.55)4.75 (0.50)4.50 (0.76)Coordination

4.50 (0.84)4.33 (1.03)4.22 (1.20)Communication4.1 Transmit CAS brief andcommunicate remarks

4.50 (0.58)4.00 (1.15)4.67 (0.58)Cooperation

4.17 (0.39)3.17 (0.41)4.00 (0.00)Coordination

4.13 (0.52)4.33 (1.03)3.89 (0.93)Communication3.2 Understand situation updates

4.33 (0.52)4.00 (0.00)3.50 (0.84)Coordination

4.44 (0.73)3.67 (0.98)3.83 (1.17)Communication3.1 Communicate options withpilot

Day 4Day 3Day 2Team cognition categoryTask


Event 1: Talk on for unauthorized drop

5How effective were FAC/others as a team?

5To what extent were team members working toward the same ends?Cooperation

4How adaptable were team members to the changing demands of thesituation?

5How effective was back-up behavior?

2How well did team members monitor each other's performance?

3How well were team members' knowledge requirements managed?Coordination

4Did closed looped communication go as expected?

3Was communication economical?

5How effective was information exchange?Communication

RatingBARSTeam cognitioncategory


Event 2: Day 4 aborted attack

5To what extent did team members display mutual trust?Cooperation




5How effective was information exchange?CommunicationAbort mission


5To what extent were team members working toward the same ends?Cooperation


5How effective was back-up behavior?





5How effective was information exchange?CommunicationTransmittalk-on

Rating BARSTeam cognitioncategory

Task


Event 3: Talk on for difficult attack


5To what extent were team members working toward the same ends?


Cooperation

5How effective was back-up behaviour?





5How effective was information exchange?Communication

RatingBARSTeam cognitioncategory


Discussion

BARS instrument captured general patterns consistentwith subjective impressions collected from observers(willingness of participants to cooperate,communications problems)

Team processes improved over the duration of theexercise for the key CAS tasks

BARS ratings captured team cognition patterns forcertain key events

Difficulty in applying BARS (number, experience)

Can’t assess interrater reliability yet

Participant skill & scenario difficulty levels madeassessing discriminant validity difficult


Take-home message

• The BARS instrument captured both micro- and macro-level team cognition patterns during the exercise

• Some of the observed patterns are directly relevant toCombat ID and fratricide avoidance

• An improved BARS instrument could support thedevelopment and assessment of training for teamcognition in CAS, thereby improving Combat ID


Next steps

• Assess inter-rater reliability, improve if needed

• Improve instrument’s ease of use

• Validate & refine BARS instrument content with SMEs

• Determine which tasks should be focused on

• Apply to more DMT CAS events with wider range oftask difficulties and participant experience

• Apply the HTA/Team cognition approach to othercollective tasks (e.g., IED detection)

team cognition during a sumulated close air support exercise cognition... · 2008. 6. 24. ·...

Documents