A Mixed Reality System for Enabling Collocated After Action

ReviewJohn Quarles

Samsun Lampotang

Ira Fischler

Paul Fishwick

Benjamin Lok

Mixed Reality for Training

• Training + Mixed Reality(MR) = Improved Training– i.e Military [x], industrial[x], medical [x]

– MR is used during the training session

• After Action Review (AAR)– Playback and review of one’s training session

• i.e. AAR in the HPS

– Currently: video reviews

– helpful for bringing trainees to competency

• AAR + MR = ?

After Action Review

• Students need:– Repetition, directed instruction, Feedback

• Educators need– Assessment Tools

• Video Based Review offers:– Assessment Tools and Repetition:

• video playback (FF,REW,etc)

– instruction and feedback:• Expert must be present during student review

Video Based AAR

• Widely used• Limitations

– Fixed, exocentric viewpoint– Minimal interaction (Play, Pause, etc)– See only what the camera sees (i.e

occlusions are problematic)– Lost information

MR Based AAR

• Video review = lost information– Goal: maximize information flow with MR

• Advantages of MR systems in general:– Egocentric viewpoint– interactivity – i.e. TUIs– Collocated virtual information (i.e. gaze)

• Can we enhance AAR with MR?

Innovations

• The Augmented Anesthesia Machine Visualization and Interactive Debriefing System (AAMVID)– MR based AAR system for Anesthesia education– Educator version: AAMVID-E– Student version: AAMVID-S

• User evaluation (19 students, 3 educators)

• General goal: Evaluate MR for use in AAR

Previous Work

• AAR [x],[x],[x]

• Recording Virtual Experiences– [x],[x],[x]

• AAR of Virtual Human Experiences– IPSViz[x]

Anesthesia Education and MR• The Augmented Anesthesia Machine [x]

– merges and abstract simulation of an anesthesia machine with a real machine (magic lens + TUI)

– Improves learning

AAR in Anesthesia Education

• Currently Video Based– Students perform a test (i.e. a fault test)– Educators and students review and critique

performance

AAR in Anesthesia Education

• MR system: AAMVID– Based on the AAM– Students (AAMVID-S):

• View their own or an expert’s interaction in situ

• Follow a directed tutorial guided by the expert’s interaction visualization

– Educators (AAMVID-E):• Can visualize and filter aggregate

student data

• AAMVID addresses video review limitations

Student AAMVID-S System

playback controls

Gaze visualizationinteraction

visualization

Student AAMVID-S System

• 3 Visualization Modes– User Review

• View what the student did

– Expert Review• View what the expert did

– Expert Tutorial• Collocates:

– expert’s gaze and interaction event boxes– Real time abstract simulation of the student’s machine

• Interaction with the real machine while mimicking expert• Hands-on experience in correcting the fault

AAMVID-S Evaluation

• Usability and performance evaluation of AAMVID-S

• 19 psychology students:

Day 1 (90 min)

Training with AAM

Day 2 (90 min)

1.Testing2. AAR of Tests with AAMVID

Day 2: Testing

• Three Fault Tests– Experimenter caused a problem in the machine

• i.e. disabled a the flow of nitrous oxide

– Participant had to decide• Whether or not a fault was present

• What the fault was (if present)

• How to correct the fault (if present)

Metrics

• Repeated Measures design:• Performance Measures:

– Participants were asked questions (each scored 0-4)• Fault absent or present• What was the fault?• How does the fault affect the patient?• How to correct the fault?

• Confidence Measures:– Self reported confidence ratings for above questions (0-

4)

Results

• Insert graphs that show significant improvements in performance and confidence.

Discussion

• Main results:– AAMVID Improved machine fault

understanding• Performance question answers improved.

– AAMVID increased confidence in fault correction ability

• Confidence scores increased

– Users prefer Expert Tutorial Mode

Educator AAMVID-E System

Aggregate Gaze Heat Mapping

Interactive Filtering(i.e. on spatial cog data)

Educator AAMVID-E System

Interactive Filtering(i.e. on spatial cog data) Interaction Graph

(markov model of class interaction)

Informal Evaluation

• 3 Experts in anesthesia education– Dr. Samsun Lampotang– David Lizdas– Dr. Nicholas Gravenstein

Observations

• Magic lens vs desktop?

• What worked and what didn’t work

Conclusions

• Presented AAMVID– Improved student understanding and

confidence– Offered needed assessment abilities to

educators that were not available before

• MR based AAR– can offer visualizations and interactivity that

video based AAR cannot