automated assessment for adaptive learning of complex tasks alan koenig markus iseli allen munro cct

Automated Assessment for Adaptive Learning of Complex Tasks

Alan Koenig

Markus Iseli

Allen Munro

CCT

Characteristics of Cognitively Complex Tasks

Often require multiple, non-trivial steps to complete

Performance can be highly variable

Often include interdependent task features

Often well suited to representation in games & simulations

– Multiple scenarios

– Low cost

Examples:

Tactical Planning Simulation Damage Control Simulation

– Low risk

– But… assessment can be difficult

Anti-Submarine Warfare

Anti-submarine warfare (ASW) is a branch of naval warfare that uses surface warships, aircraft, or other submarines to find, track and deter, damage, or destroy enemy submarines.

– Carried out by teams of people within and across ships

– Requires synthesis of large quantities of information in short periods of time

– Heavy emphasis on situation awareness and decision making skills (especially among TAO’s)

Assessment Methodology Overview

ASW Ontology Construction

ExpertsRules /Doctrine

Procedures /Processes

Facts /Observations

DOMAIN

Ontology Excerpt

Bayesian Network Development

OntologyNodes

Directionalityof Links

Observables /SME Input

Strength of Relationships

Characterizing the nature of relationships is essential

Observable Actions

Latent Variables

SonarPlanning

PassiveSonar Planning

ActiveSonar

Planning

UseShipPassiveSonar DropPassiveSonobuoys

DropActiveSonobuoys UseDippingSonar UseShipActiveSonar

ASW Skilled

Torpedo CounterMeasures

MEUProtectionSub Hunting

Plan for Opposition

Intell on Opposition

Opposition Estimation

ShowMissionInfo ShowBriefing

EstLLSubSpeed

EstLLTorpedoRange

0.5

SonarPlanning

TDA AvoidManeuvers

PassiveSonar Use

ActiveSonar Skill

AlterCourseTo AvoidFast

AlterCourseTo AvoidSlow

UseShipPassiveSonarDropPassiveSonobuoy

s

DropActiveSonobuoys UseDippingSonar UseShipActiveSonar

TDA Avoidance

DontEverEnterTDALimLines

Estimation

EstDatumSubSpeed EstDatumTorpedoRange

Datum Estimation

TorpedoDeception

TorpedoEscape

NixieDropped MICDropped

TorpedoEvasion

TorpedoCounter

Bayesian Network for TDA Avoidance

ASW Skilled

Torpedo CounterMeasures

MEUProtectionSub Hunting

Plan for Opposition

Intell on Opposition

Opposition Estimation

0.5

TDA AvoidManeuvers

DropActiveSonobuoys UseDippingSonar UseShipActiveSonar

TDA Avoidance

LimLines Estimation

Datum Estimation

TorpedoDeception

TorpedoEscape

Bayesian Network for TDA Avoidance

ShowMissionInfo ShowBriefing

EstLLSubSpeed

EstLLTorpedoRange

SonarPlanning

PassiveSonar Use

ActiveSonar Skill

AlterCourseTo AvoidFast

AlterCourseTo AvoidSlow

UseShipPassiveSonarDropPassiveSonobuoy

s

DontEverEnterTDA

EstDatumSubSpeed EstDatumTorpedoRange

NixieDropped MICDropped

TorpedoEvasion

TorpedoCounter

Conditions for TDA Avoidance to be considered by Bayesian network

Observable Actions

Latent Variables

The Advantage of Bayesian Networks

SonarPlanning

PassiveSonar Planning

ActiveSonar

Planning

UseShipPassiveSonar DropPassiveSonobuoys

DropActiveSonobuoys UseDippingSonar UseShipActiveSonar

Probability of mastery of the latent variables is inferred from the scored observable actions

P(ASP): Probability of mastery of concept Active Sonar Planning

P(UDS | ASP): Conditional probability. Probability of mastery of concept UseDippingSonar, given information about mastery of concept Active Sonar Planning

Example: Determining probabilities

P(UDS):Probability of mastery of concept UseDippingSonar

P(ASP) x P(UDS | ASP)

P(UDS)P(ASP | UDS) =

Real-Time Formative Assessment

•Provide performance feedback

•Provide practice / resources

•Add / change tasks

•Add / change affordances

FORMATIVE ASSESSMENT

Simulation(Sandbox)

ADAPTIVE OPTIONS

Observable Actions & Events

CAA(with Bayesian Network)

Check the Briefing—Good!

1st PROBLEM: Chinese Kilos Threat Problem

00:00:00 .k.Announce. ShowBriefing 0.95

The fact that we’re up against Kilos implies a silent sub speed of 6 Kt.and a torpedo range of 12,000 yds.

Failed to Correct Estimates—Bad!

1st PROBLEM: Chinese Kilos Threat Problem

07:26:32 .k.Announce. EstSubSpeed 0.207:26:32 .k.Announce. EstTorpedoRange 0.2

The estimated sub speed was left at 3 Kt. So the limit lines were not accurate, relative to the briefing.

Scored at end, because these values were never changed.

Failed to Correct Estimates—Bad!

1st PROBLEM: Chinese Kilos Threat Problem

07:26:32 .k.Announce. DontEverEnterTDA 0.1

Uh-oh! The MEU entered the TDZ.

17

Problem 1

18

Problem 1

19

Problem 1

20

Problem 1

21

Problem 1

Check the Briefing—Good!

2nd PROBLEM: Threat West of Gibraltar

This time the mission brief gives specific estimates of hostile sub speed and torpedo range.

00:00:00 .k.Announce. ShowBriefing 0.95

Datum Estimates2nd PROBLEM: Threat West of Gibraltar

The datum estimates forsub speed and torpedorange don’t match the briefing.

Datum Estimates Corrected—Good!

2nd PROBLEM: Threat West of Gibraltar

Set to the correct values.

01:14:03 .k.Announce. EstSubSpeed 0.9501:14:03 .k.Announce. EstTorpedoRange 0.95 Scored at end.

Check Limit Lines2nd PROBLEM: Threat West of Gibraltar

Perhaps not a wide enough berth…

Plot a New Course2nd PROBLEM: Threat West of Gibraltar

A safer course?

Check Limit Lines Again2nd PROBLEM: Threat West of Gibraltar

Looks safer

Successful Avoidance2nd PROBLEM: Threat West of Gibraltar

Success!

01:14:03 .k.Announce. DontEverEnterTDA 0.95

29

Problem 2

30

Problem 2

31

Problem 2

32

Problem 2

33

1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Problem 1Problem 2

1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Problem 1Problem 2

Plan for Opposition

TDA Avoidance

Sub speed: Wrong estimate

Entered TDA

Prob

abili

ty o

f Mas

tery

Prob

abili

ty o

f Mas

tery

time

time

ASW Skilled

Torpedo CounterM.

MEUProtectionSub Hunting

Plan for Opposition

Intelligence on Opposition

Opposition Estimation

GetMissionInfo GetBriefing EstSubSpeed EstTorpedoRange

SonarPlanning

TDA AvoidManeuvers

SenseQuietly

SenseLoudly

ManeuverFastManeuverSlow

ShipsPassivSonar PassiveSonobuoys ActiveSonobuoys DippingSonar ShipsActivSonar

TDA Avoidance

B20: score 0.95D20: score 0.6F20: score 0.1

B16: score 0.95D16: score 0.7F16: score 0.05 B5: score 0.9

D5: score 0.7B6: score 0.95D6: score 0.7

B11: score 0.95D11: score 0.8F11: score 0.2

B12: score 0.95D12: score 0.8F12: score 0.2

B36: score 0.9E36: score 0.3F36: score 0.2

B40: score 0.95C40: score 0.4E40: score 0.3

C44: score 0.9D44: score 0.6F44: score 0.05

B28: score 0.95D28: score 0.6

B32: score 0.95E32: score 0.3

DontEnterTDAIfPos

B24: score 0.95C24: score 0.9F24: score 0.1

TDA avoidanceBayesian network sub-net

Initial Evaluation Feedback

CAA Debugging OutputInstructional Feedback High Level Assessments

First Successful Problem

Into the Next Problem

Adaptive Problem Selection

A new problem is advised.

Loading the Problem

The advised problem is selected

Initial Actions in the New Problem

Preparing to Find the Sub

Launching a helicopter

Detecting the Sub

Sonobuoys detect sub

Recording of Performances & Assessments

Simulation(Sandbox)

Log Files(Actions and Events)

These performance recordscan be replayed.

Report Generation(CAA Monitor)

Assessment History(Per Student)

Report

CAA

Questions / Comments?

CCT