Measuring Team Shared Understanding: Using Analysis-

Constructed Shared Mental Model Methodology

Measuring Team Shared Understanding: Using Analysis-

Constructed Shared Mental Model MethodologyTristan E. Johnson

Learning Systems Institute, Florida State University, Tallahassee, FL, USA

tjohnson@lsi.fsu.edu

International Workshop and Mini-conference onExtending Cognitive Load Theory and Instructional Design

to the Development of Expert Performance

August 29-30, 2005 Open University of the Netherlands

BackgroundBackground1. Team Performance

• Team Cognition Link between SMM and Team Performance

2. Shared Understanding and Shared Mental Models

Development of SMM and its relation to team performance

Team CognitionTeam Cognition

Elaborated view of team cognition including team interactions and SMM development

Shared Knowledge TypesShared Knowledge Types

1. Task Knowledge—domain specific2. Team Knowledge—5 factors

Team Knowledge Factors Team Knowledge Factors 1. Team Knowledge

• Knowledge about team members and tasks that they need to perform• Teammates knowledge, Task knowledge

2. Team Skills• Abilities associated with successful job performance• Communication skills, Interpersonal skills, Leadership skills, Skills to

deal with conflict and team cohesion

3. Team Attitudes• Internal state that influences team members’ choices or decision to

act in a certain way under particular circumstances • Shared belief, Shared value

4. Team Dynamics• Combination of dynamic processes of team coordination and team

cohesion• Team coordination, Team cohesion

5. Team Environment• External conditions affecting the foundation of the team mental model• Technology, Organization, Synchrony & Geographic dispersion

Measuring Task KnowledgeMeasuring Task Knowledge1. Measuring Shared Understanding—

measuring concept relatedness • Card sorting, cognitive interviewing, MDS,

Pathfinder, surveys, casual maps (Langan-Fox, Code, Langfield-Smith, 2000; Trochim, 1989)

• Concept Mapping Statistical analysis Descriptive analysis

2. Analysis Constructed - Shared Mental Model (AC-SMM)

SMM Elicitation TechniquesSMM Elicitation Techniques1. TmC-SMM—Whole team elicitation (1 map)2. AC-SMM—Individual elicitation with aggregation (n maps)

SMM i— desired shared mental model stateTmC-SMM — involves team negotiation and interactionSMM∂— altered team shared mental model stateAC-SMM —retains the initial ICMM state

AC-SMM Methodology RationaleAC-SMM Methodology Rationale1. Knowledge Elicitation

• Process allows simultaneous consideration of concepts• Reflection and changes during elicitation

2. Analysis• Allows for explication of implicit relationships—

considering 1) logic, 2) structure, and 3) spatial orientation

3. Relatedness• Specific to three levels

Concepts Links Clusters

4. Appropriate for studying shared understanding in applied settings

AC-SMM Methodology Overview AC-SMM Methodology Overview 1. Instrument Design

• Structured/Semi-Structured/Unstructured• Task Analysis (Generate Concepts)

2. Data Collection• Guided Practice • Individually Constructed Mental Model (ICMM) Elicitation

3. Data Analysis• Phase I: ICMM Analysis/Coding

Relatedness at concepts, links, clusters levels Allows for explication of implicit relationships Implicit coding has [logic and spatial] or [logic and structural]

support• Phase II: Shared Analysis

Determine sharedness level—number or percentage of team members

• Phase III: AC-SMM Construction Generates SMM

Phase I: ICMM AnalysisPhase I: ICMM AnalysisFactor 1: Concepts

• Explicit individual nodes

Factor 2: Links• Two concepts joined explicitly [connector] or

implicitly

Factor 3: Clusters• Two or more connectors explicitly bridging three

or more concepts• May have implicit connections with evidence• Combination of clusters—Sub- and Super-

clusters

Factor 4: Emphasis and Sequence• Explicit notation of node emphasis or node order

ICMM Coding—LinksICMM Coding—Links

ICMM Coding—ClustersICMM Coding—Clusters

ICMM Coding—Emphasis and SequenceICMM Coding—Emphasis and Sequence

ICMM Coding ExampleICMM Coding Example

Phase II: Shared AnalysisPhase II: Shared Analysis1. Determine Sharedness Level Criterion—Number or

Percentage2. Shared Data Used for AC-SMM Construction

Phase III: AC-SMM ConstructionPhase III: AC-SMM Construction

ResearchResearch1. General Research Focus

• What task knowledge is shared?• How does shared understanding change over

time?• What are the patterns of change? • What is the affect of task performance on the

shared understanding of the team?

Timeline of Data Collection and Hours Worked During PQS Workshop Sessions

Day 1 Day 2 Day 3 Day 4 Workshop

(C) a Data (#) b Hrs Hrs

Data (#) b Hrs Hrs

Data (#) b Hrs Hrs

Data (#) b

Total Hrs

Workshop 1 (3)

Pre (4) 0.00 3.50 None 0.00 0.00

Mid (4) 1.08 1.88 Post (3) 6.47

Workshop 2 (10) Pre (11) 4.25 8.00 None 0.00 0.00 Mid (11) 4.75 2.00 Post (10) 19.00

Workshop 3 (10) Pre (12) 6.00 1.67

Mid (13) 3.67 4.87 None 5.00 4.00 Post (15) 25.21

Workshop 4 (7) Pre (13) 7.83 7.75 Post (7) 0.00 0.00 None 0.00 0.00 None 15.58

Note. a C = Number of team members submitting complete datasets. b # = Number of team members submitting ICMM for data collection period.

Data Collection TimelineData Collection Timeline

Concepts: [A] - Fleet notification of change [L] - Section 300 - Watchstations [B] - Analysis - Determine level of change [M] - Content revisions [C] - Verify corrections [N] - Add line item/task [D] - Model manager [O] - Add new references [E] - Subject matter expert [P] - Formatting [F] - Non-content related revisions [Q] - Keep line item/task [G] - Consensus [R] - Summary of changes [H] - Ve rify existing references [S] - Referencing [I] - Facilitator [T] - Delete line item/task [J] - Section 100 - Fundamentals [U] - Sequencing [K] - Section 200 - Systems [V] - Semantics (wording)

ConceptsConcepts

Team Profiles FindingsTeam Profiles Findings

Profile of Workshop 1 PQS Team

Like / Dislike Concept Mapping b

Participant by Role

PQS Workshop

Exp a

Familiar with

Others

Previous Concept

Mapping Exp

a Pre- task Post-task

Year s in Navy

FAC 01 Yes No Yes 7.00 6.00 17.00 FAC 02 c Yes No No -- -- 22.00 SME 01/MM No Yes No -- 5.00 17.00 SME02 No Yes No -- 8.00 17.00 Average All Workshop Participants 7.00 6.33 18.25 Average Complete Datasets 7.00 6.33 17.00 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicates participant(s) did not submit all data. -- Indicates no response.

Profile of Workshop 2 PQS Team

Like / Dislike Concept Mapping b

Participant by Role

PQS Workshop

Exp a

Familiar with

Others

Previous Concept

Mapping Exp

a Pre- task Post-task

Year s in Navy

FAC 01 Yes No No -- 5.00 18.00 SME 01/ MM No Yes No -- 5.00 20.00 SME 02 c No Yes No 5.00 6.00 SME 03 No Yes No -- 7.00 19.00 SME 04 No Yes Yes 5.00 9.00 19.00 SME 05 Yes Yes No -- 5.00 19.00 SME 06 -- -- No -- 8.00 15.00 SME 07 Yes Yes No -- 0.00 17.00 SME 08 No Yes No -- -- 11.00 SME 09 No -- No -- -- 10.00 SME 10 No No No -- 7.00 18.00 Average All Workshop Participants 5.00 5.75 15.64 Average Complete Datasets 5.00 5.75 16.60 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicates participant(s) did not submit all concept maps. -- Indicates no response

Profile of Workshop 3 PQS Team

Like / Dislike Concept Mapping b Participant

by Role

PQS Workshop

Exp a

Familiar with

Others

Previous Concept Mapping

Exp a Pre- task Post-task

Years in Navy

FAC 01 Yes No Yes 6.00 5.00 19.00 SME 01 /MM c Yes Yes Yes 6.00 5.00 18.00 SME 02 c Yes Yes No -- -- 26.00 SME 03 c No Yes No -- -- 23.00 SME 04 c No Yes No -- 6.00 19.00 SME 05 No Yes No -- 0.00 12.00 SME 06 Yes Yes No -- -- 26.00 SME 07 No Yes No -- 7.00 12.00 SME 08 c No Yes No -- 8.00 14.00 SME 09 c No Yes No -- 7.00 8.50 SME 10 c No No No -- 8.00 16.00 SME 11 c No -- No -- -- 7.00 SME 12 No -- No 8.00 10.00 4.00 SME 13 -- No No -- -- 3.00 SME 14 -- -- No -- -- 2.50 SME 15 No No No -- 7.00 15.00 SME 16 -- -- -- -- -- -- SME 17 -- -- -- -- 9.00 -- SME 18 -- -- -- -- -- -- Average All Workshop Participants 6.67 6.55 14.06 Average Complete Datasets 6.00 6.50 16.72 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicates participant(s) did not submit all concept maps. -- Indicates no response

Profile of Workshop 4 PQS Team

Like / Dislike Concept Mapping b

Participant by Role

PQS Workshop

Exp a

Familiar with

Others

Previous Concept Mapping

Exp a Pre- task Post-task

Year s in Navy

FAC 01 Yes Yes Yes 5.00 1.00 22.00 SME 01/MM c Yes Yes Yes 5.00 -- 21.00 SME 02 No Yes No -- 5.00 19.00 SME 03 c No Yes No -- -- 27.00 SME 04 Yes Yes No -- 6.00 8.00 SME 05 c No Yes No -- -- 19.00 SME 06 c No Yes No -- -- 19.50 SME 07 Yes Yes Yes 7.00 8.00 25.00 SME 08 c No -- No -- -- 14.00 SME 09 No Yes No -- -- 21.00 SME 10 c -- Yes No -- -- 23.00 SME 11 -- -- -- -- -- -- SME 12 -- -- -- -- -- 24.00

Average All Workshop Participants 5.67 5.00 20.21 Average Complete Datasets 6.00 5.00 19.60 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicate participant(s) did not submit all ICMMs. -- Indicates no response.

Shared Data Findings, Team 1 Only Shared Data Findings, Team 1 Only

Shared Data (³ 5 0%) Ğ Workshop 1

Workshop 01 Complete Datasets ICMM Pre-task Mid-task Post-task

# Team Members Submitting 3 3 3 Component # % # % # %

S = Shared Across ICMMs

Concepts

[A] Ğ Fleet notification of change 3 100% 2 67%

[B] Ğ Analysis-Determine level of change 3 100% 2 67%

[C] Ğ Verify corrections 3 100% 3 100% 3 100% S

[D] Ğ Model manager 3 100% 2 67% 3 100% S

[E] Ğ Subject matter expert 3 100% 2 67% 3 100% S

[F] Ğ Non-content related revisions 3 100% 2 67%

[G] Ğ Consensus 3 100% 2 67% 3 100% S

[H] Ğ Ve rify existing references 3 100% 2 67% 3 100% S

[I] Ğ Facilitator 3 100% 3 100% 3 100% S

[J] Ğ Section 100 Ğ Fundamentals 3 100% 2 67% 3 100% S

[K] Ğ Section 200 Ğ Systems 3 100% 2 67% 3 100% S

[L] Ğ Section 300 Ğ Watchstations 3 100% 2 67% 3 100% S

[M] Ğ Content revisions 3 100% 3 100% 2 67% S

[N] Ğ Add line item/task 3 100% 3 100% 3 100% S

[O] Ğ Add new references 3 100% 2 67% 3 100% S

[P] Ğ Formatting 3 100% 2 67% 3 100% S

[Q] Ğ Keep line item/task 3 100% 3 100% 3 100% S

[R] Ğ Summary of changes 3 100% 2 67%

[S] Ğ Referencing 3 100% 2 67% 3 100% S

[T] Ğ Delete line item/task 3 100% 3 100% 3 100% S

U Ğ Sequencing 3 100% 3 100% 3 100% S

V Ğ Semantics (wording) 3 100% 3 100% 2 67% S

Shared Data (³ 5 0%) Ğ Workshop 1 Workshop 01 Complete Datasets ICMM Pre-task Mid-task Post-task # Team Members Submitting 3 3 3 Component # % # % # %

S = Shared Across ICMMs

Directional Links

[D>A] 2 67%

[I>D] 2 67%

[I>E] 2 67%

[I>J] 2 67%

[J>C] 2 67%

[K>J] 2 67%

[L>K] 2 67%

[M>F] 2 67%

[P>L] 2 67%

Directional Links [U>V] 2 67%

Non-Directional Links

[E,I] 2 67%

[J,K] 2 67%

[K,L] 2 67%

[N,Q] 2 67%

[Q,T] 2 67%

Shared Data Summary Per TeamShared Data Summary Per Team

Shared Data Summary - Workshop 1

Workshop 01 Complete Datasets ICMM Datasets Pre-task Mid-task Post-task

# Team Members Submitting 3 3 3 Concepts 29 21 24 Shared (50% or more) 22 20 21 % Shared 75.86% 95.24% 87.50% Sequence Components 0 3 1 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Clusters 17 18 22 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Directional Links 68 59 52 Shared (50% or more) 3 3 4 % Shared 4.41% 5.08% 7.69% Non-Directional Links 32 17 19 Shared (50% or more) 1 2 2 % Shared 3.13% 11.76% 10.53% Important Concepts 0 6 4 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Total Components 146 124 121 Shared (50% or more) 26 25 27 % Shared 17.81% 20.16% 22.31%

Shared Data Summary - Workshop 2

Workshop 02 Complete Datasets ICMM Datasets Pre-task Mid-task Post-task

# Team Members Submitting 10 10 10 Concepts 26 23 22 Shared (50% or more) 22 22 22 % Shared 84.62% 95.65% 100.00% Sequence Components 8 0 0 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Clusters 54 56 45 Shared (50% or more) 0 1 1 % Shared 0.00% 1.79% 2.22% Directional Links 108 98 93 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Non-Directional Links 79 84 83 Shared (50% or more) 1 0 2 % Shared 1.27% 0.00%- 2.41% Important Concepts 13 7 5 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Total Components 290 268 248 Shared (50% or more) 23 23 25 % Shared 7.93% 8.58% 10.09%

Shared Data Summary (³ 50%) Ğ Workshop 3

Workshop 03 Complete Datasets ICMM Datasets Pre-task Mid-task Post-task

# Team Members Submitting 10 10 10 Concepts 24 22 22 Shared (50% or more) 21 21 21 % Shared 87.50% 95.45% 95.45% Sequence Components 3 0 0 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Clusters 56 53 53 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Directional Links 142 129 129 Shared (50% or more) 3 1 4 % Shared 2.11% 0.78% 3.10% Non-Directional Links 30 39 42 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Important Concepts 7 17 3 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Total Components 262 260 251 Shared (50% or more) 24 22 25 % Shared 9.16% 8.46% 9.96%

Shared Data Summary (³ 50%) Ğ Workshop 4

Workshop 4 Complete Datasets ICMM Datasets Pre Mid Post

# Team Members 7 0 7 Concepts 25 -- 24 Shared (50% or more) 19 -- 16 % Shared 76.00% -- 66.67% Sequence Components 8 -- 10 Shared (50% or more) 0 -- 0 % Shared 0.00% -- 0.00% Clusters 39 -- 44 Shared (50% or more) 0 -- 1 % Shared 0.00% -- 2,27% Directional Links 128 -- 45 Shared (50% or more) 0 -- 0 % Shared 0.00% -- 0.00% Non-Directional Links 53 -- 127 Shared (50% or more) 2 -- 1 % Shared 3.77% -- 0.79% Important Concepts 2 -- 0 Shared (50% or more) 0 -- 0 % Shared 0.00% -- -- Total Components 255 -- 250 Shared (50% or more) 21 -- 18 % Shared 8.24% -- 7.20%

Cross Case FindingsCross Case Findings

Concepts from ACSMMs Shared ³ 50% as Compared across Workshops

ACSMM Pre-Task Mid-Task Post-Task Workshop 1 2 3 4 1 2 3 1 2 3 4 Members 3 10 10 7 3 10 10 3 10 10 7

Concept % % % % % % % % % % % [A] 100 100 100 71 0 100 100 67 100 90 57 [B] 100 100 100 100 0 100 90 67 90 80 71 [C] 100 100 100 71 100 90 90 100 100 80 86 [D] 100 90 80 86 67 100 90 100 100 90 57 [E] 100 100 100 86 67 100 100 100 100 100 71 [F] 100 80 0 0 67 70 0 0 50 0 0 [G] 100 80 100 71 67 90 90 100 100 90 86 [H] 100 80 100 100 67 90 100 100 100 100 86 [I] 100 100 90 71 100 100 100 100 100 90 0 [J] 100 80 100 71 67 90 100 100 90 100 71 [K] 100 80 100 71 67 90 100 100 90 100 71 [L] 100 80 100 71 67 90 100 100 90 100 71 [M] 100 100 60 71 100 80 60 67 70 70 57 [N] 100 80 90 71 100 70 90 100 70 90 57 [O] 100 80 100 100 67 90 100 100 100 100 86 [P] 100 100 70 86 67 80 80 100 90 80 57 [Q] 100 90 80 71 100 90 80 100 90 90 57 [R] 100 100 100 86 0 90 100 67 100 80 0 [S] 100 100 80 57 67 100 70 100 100 90 0 [T] 100 60 90 71 100 60 90 100 70 90 57 [U] 100 80 80 0 100 70 60 100 70 60 0 [V] 100 70 80 0 100 70 80 67 70 90 0

ACSMM Scores per Wo rkshop

ACSMM Factor Workshop 1 Workshop 2 Workshop 3 Workshop 4

Pre-task ACSMM Score (Points) Concept 22 22 21 19 Sequence 7 9 9 4 Cluster 10 15 15 15 Link 20 20 18 8 Important Concept 0 0 0 0 Total ACSMM 59 66 63 46

Post-task ACSMM Score (Points)

Concept 21 22 21 16 Sequence 7 8 8 4 Cluster 15 25 20 20 Link 20 14 22 12 Important Concept 0 0 0 0 Total ACSMM 63 69 71 52

Change in Score from Pre-task to Post-task (Points)

Concept -1 0 0 -3 Sequence 0 -1 -1 0 Cluster 5 10 5 5 Link 0 -6 4 4 Important Concept 0 0 0 0 Total ACSMM 4 3 8 6

Pre, Mid, Post AnalysisPre, Mid, Post Analysis

ACSMM ScoresACSMM Scores

ACSMM Scores and Team Performance Outcomes

Measures of SMM and Team Performance Workshop 1 Workshop 2 Workshop 3 Workshop 4 Pre-Task ACSMM Score 59 66 63 46 Post-Task ACSMM Score 63 69 71 52 Change in ACSMM Score from Pre to Post 4 3 8 6 % Change from Pre to Post 6.78% 4.55% 12.70% 13.04% Hours Worked 6.47 19.00 25.21 15.58 Major Revisions 103 180 285 219 Major Revisions as % of Al l Revisions 41.04% 26.05% 13.08% 75.26% Minor Revisions 148 511 1894 72 Minor Revisions as $ of All Revisions 58.96% 73.95% 86.92% 24.74% Total Revisions 251 691 2179 291 Revisions per Hour 38.79 36.37 86.43 18.68

General FindingsGeneral Findings

1. Similarity among ICMMs tends to increase as does the number of clustered concepts, the tendency is for the number of concepts used to decrease.

2. ICMMs were becoming more structured and more representative of the team task

3. These ideas are not yet proven. 4. We have designed a set of studies to try and

validate our hypothesis5. This work is intended to not only learn about

teams that work in the various settings, but to validate the AC-SMM analysis model

SummarySummary1. Richer description of shared understanding in

teams2. AC-SMMs compared over time to determine

change in shared understanding3. Lacks weighted measures and precise distances

between concepts, but future work will include descriptive statistics of the key factors

4. Lack of prepositional descriptors5. As we become more precise and descriptive we

can utilize this new knowledge to better explain and understand team cognition

6. Facilitate team training with intent to improve team performance outcomes

Thanks for your attention.

Questions?

Thanks for your attention.

Questions?

Findings and Extra SlidesFindings and Extra Slides

Findings Across Pre, Mid, PostFindings Across Pre, Mid, Post

Participants & ContextParticipants & Context

1. Participants• Personnel Qualification Standards (PQS) Team• Team Task • Team Members • Team Member Roles

2. Context• U.S. Navy Training Center, Pensacola, FL• Face to face workshop• Equipment

Findings From AC-SMM AnalysisFindings From AC-SMM Analysis

• What is shared?• Does shared understanding change over time?

Secondary AnalysisSecondary Analysis

1. Sequence• Where concepts were placed within each ICMM• Focus on key concepts

Team member roles Sections of PQS book Referencing Questions

2. Links• Relationships between concepts without directionality

3. Clusters• Relationships between concepts without directionality• Commonalities between related clusters of concepts

Secondary AnalysisSecondary Analysis

• Links - Relationships between concepts without directionality • Clusters - Relationships between concepts without

directionality and commonalities between related clusters of concepts

Example of Cluster

[[B,D], [B,E], [B,I]]

with Secondary Clusters [B,D,E,I] and [D,E,I]

without Related Links

[D,E], [D,I], and [E,I]

All Data, Shared by ≥ 2

All Data, Shared by ≥ 50%

Follow-up AnalysisFollow-up Analysis

1. Sequence• Started with original data submitted by each team member• Where concepts were placed within each ICMM• Focus on key concepts

Team member roles (concepts [D], [E], [I])

Sections of PQS book (concepts [J], [K], [L])

Referencing (concepts [H], [O], [S])

Questions (concepts [N], [Q], [T])

All Data, Shared by ≥ 2Secondary Analysis

All Data, Shared by ≥ 50%Secondary Analysis

Complete Datasets, Shared by ≥ 50%

Complete Datasets, Shared by ≥ 2Secondary Analysis

Complete Datasets, Shared by ≥ 50%Secondary Analysis