© Silvia Miksch

AsgaardAsgaardAsgaardAsgaard / AsbruAsbruAsbruAsbru Project :Designing Task-Specific Problem-Solving Methods to Supportthe Design and the Execution of Time-Oriented Skeletal Plans

http://www.asgaard.tuwien.ac.at

Peter JohnsonSilvia MikschYuval ShaharStanford University

Stanford Medical Informatics(SMI)

International Cooperation• Vrije Universiteit Amsterdam• University of Aberdeen• University of Augsburg• CBO, Dutch - Healthcare Improvement

Klaus HammermüllerRobert Kosara

Andreas Seyfangand students

Vienna University of TechnologyVienna, Austria

Yuval Shaharand students

Ben-Gurion University of the NegevBeer-Sheva, Israel

© Silvia Miksch

Outline

• Motivation– Medical Therapy Planning– Plan Representations Limitations

• The Asgaard/Asbru Project• Asbru Language

– Terminology– Knowledge Roles

• AsbruView– Metaphor Graphics

• Evaluation, Conclusion

New LanguageAsbru

© Silvia Miksch © Silvia Miksch

© Silvia Miksch

Medical Therapy Planning• Clinical Protocols• Current Representations of Protocols

– Free Text– Tables– Flow-Charts

© Silvia Miksch

Acute Severe Asthma GuidelineIndications:

- Cannot complete sentences- Pulse > 110 beats/minute- Respiration > 25 breaths/minute- Peak flow < 50% of predicted or best

Therapy:• Oxygen 40-60% if available• Nebulized salbutamol 5mg or nebulized terbutaline 10mg• Oral prednisolone 30-60 mg or i/v hydrocortisone 200mg• Monitor responses 15-30 minutes after nebulisation

If any signs of acute asthma persist: +Arrange hospital admission +While awaiting ambulance: repeat nebulised beta2 and give with nebulised ipratropium 500 micrograms OR give subcutaneous terbutaline OR give slow i/v aminophylline 250mg Else if symptoms have improved, step up usual treatment; follow up

© Silvia Miksch © Silvia Miksch

Medical Therapy Planning• Clinical Protocols• Current Representations of Protocols

– Free Text– Tables– Flow-Charts

• Used for– Communication– Quality Assessment

© Silvia Miksch

Plan Representation

• Huge volume of data

• Pre- and post-conditions are needed to controlthe execution of durative actions or plans

• Goal may not be achievable in time

• Sequential, parallel, and cyclical execution ofplans is necessary

requirements

© Silvia Miksch

Plan Representation• Large domain knowledge is available

� Incomplete and non-deterministic information

• Planning agent is not omniscient�Unobservable underlying processes�Goals are not categorical and unchanged over time

• Everything is durative, not instantaneous• Multiple time lines based on different granularities• Plans can be suspended• Interleaving of plan design and execution need

missing requirements

© Silvia Miksch

AsgaardAsgaardAsgaardAsgaard / AsbruAsbruAsbruAsbru Concept

& TASKS

INPUTTime-Oriented

Sharable Plan-Specification

Library

OUTPUTTime-Oriented

Explanations

??

Clinical Guidelines

Raw Data

Patient Data

Raw Data

Task-SpecificPSM Abstraction

Visualization

Plan Selection - Monitoring Critiquing Plan Execution

Future Projection Evaluation

Verification Validation

© Silvia Miksch

Asbru‘s Key Features

• Hierarchical Decomposition of Plans• Temporal Annotations & Uncertainty• Knowledge Roles

– Preferences– Intentions– Conditions– Effects– Plan Layouts

© Silvia Miksch

Hierarchical Decomposition of Plans

Time

Plan

s

PLAN AA

PLAN A1

A

C D

EF

B

EE

PLAN A2

G

H IE

© Silvia Miksch

Parameter Proposition• Parameter or its Abstraction• Value• Context• Time Annotation

Example: ”Temperature in the context of controlled ventilation

is high for more than 3 hours starting from 24 to 26 hoursafter I-RDS was diagnosed.”

(Temperature High Controlled-Ventilation [[24 HOURS, 26 HOURS], [_, _], [3 HOURS,_], I-RDS-diagnosed])

© Silvia Miksch

Asbru 's Time Annotation

Reference ESS LSS EFS LFS

Definition: [[ESS, LSS], [EFS, LFS], [MinDu, MaxDu], Reference]

MaxDu

MinDu

Time

Example: [[_, _], [_, _], [180 MIN, _], *self*]

undef. undef. undef. undef.

180 Min

*self*

undef.

Time

ESS Earliest Starting ShiftLSS Latest Starting Shift

EFS Earliest Finishing ShiftLFS Latest Finishing Shift

MinDu Minimum DurationMaxDu Maximum Duration

Reference Point

© Silvia Miksch

Effects

Arg-dependency

Overall

Asbru‘s Knowledge Roles (1/2)

Intentions

Intermediate state

Intermediate action

Overall state pattern

Overall action pattern

Preferencesstrategy

...

Conditions ...

Plan body

Sequential

Cyclical

Concurrent

Parallel

Unorder

Any Order

© Silvia Miksch

Asbru‘s Knowledge Roles (2/2)

Conditions

Filter

Setup

Suspend

Abort

Complete

ReactivateActivate

© Silvia Miksch

I-RDS Example in AsbruAsbruAsbruAsbru 6.5(PLAN controlled-ventilation(PREFERENCES (SELECT-METHOD BEST-FIT))(INTENTION:INTERMEDIATE-STATE (MAINTAIN STATE(BG) NORMAL controlled-ventilation *))(INTENTION:INTERMEDIATE-ACTION (MAINTAIN STATE(RESPIRATOR-SETTING) LOW controlled-

ventilation *))(SETUP-PRECONDITIONS (PIP (<= 30) I-RDS *now*)

(BG available I-RDS [[_, _], [_, _],[1 MIN,_](ACTIVATED initial-phase#1)]))(ACTIVATED-CONDITIONS AUTOMATIC)(ABORT-CONDITIONS ACTIVATED

(OR (PIP (> 30) controlled-ventilation [[_, _], [_, _], [30 SEC, _], *self*]) (RATE(BG) TOO-STEEP controlled-ventilation [[_, _], [_, _], [180 MIN,_], *self*])))(SAMPLING-FREQUENCY 10 SEC))

(COMPLETE-CONDITIONS(FiO2 (<= 50) controlled-ventilation [[_, _], [_, _], [180 MIN, _], *self*])(PIP (<= 23) controlled-ventilation [[_, _], [_, _], [180 MIN, _], *self*])(f (<= 60) controlled-ventilation [[_, _], [_, _], [180 MIN, _], *self*])(patient (NOT DYSPNEIC) controlled-ventilation [[_, _], [_, _], [180 MIN, ], *self*]))(STATE(BG) (OR NORMAL ABOVE-NORMAL) controlled-ventilation [[_, _], [_, _], [180 MIN,_], *self*])

(SAMPLING-FREQUENCY 10 MIN))(DO-ALL-SEQUENTIALLY

(one-of-increase-decrease-ventilation)(observing)))

© Silvia Miksch

I-RDS Example in AsbruAsbruAsbruAsbru 7.1d<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE plan-library (View Source for full

doctype...)>- <plan-library>- <domain-defs>- <domain name="i-rds">- <record-def name="patient"><field-def name="name" type="STRING" /><field-def name="sex" type="maleorfemale" /></record-def>- <qual-scale-def name="rate-change"><qual-entry entry="normal" /><qual-entry entry="too-steep" /></qual-scale-def>- <qual-scale-def name="bg_state"><qual-entry entry="normal" /><qual-entry entry="strange" /><qual-entry entry="critical" /></qual-scale-def>- <qual-scale-def name="respirator-setting"><qual-entry entry="low" /><qual-entry entry="normal" /><qual-entry entry="high" /></qual-scale-def>- <qual-scale-def name="maleorfemale"><qual-entry entry="male" /><qual-entry entry="female" /></qual-scale-def>- <qual-scale-def name="facecolor"><qual-entry entry="not_dypneic" /><qual-entry entry="dyspneic" /></qual-scale-def>- <unit-def name="frequency" rounding="nearest">- <compound-def><numerator />- <denominator>

<unit-class name="time" exponent="1" /></denominator></compound-def></unit-def>- <unit-def name="flow-unit" rounding="nearest">- <compound-def>- <numerator><unit-class exponent="1" name="amount" /></numerator><denominator /></compound-def></unit-def>- <parameter-def name="bg" required="yes"

type="bg_state">- <qual-parameter-def scale="bg_state">- <limits><context-ref name="controlled-ventilation" /><limit-entry value="1" /><limit-entry value="5" /><limit-entry value="7" /><limit-entry value="9" /><numerical-constant unit="s" value="30" /></limits><raw-data-def mode="manual" /></qual-parameter-def><numerical-constant unit="s" value="10" /></parameter-def>- <parameter-def name="repirator-setting" required="no"

type="respirator-setting">- <qual-parameter-def scale="respirator-setting">- <limits><context-ref name="i-rds" /><limit-entry value="low" /><limit-entry value="normal" /><limit-entry value="high" />

<numerical-constant unit="s" value="10" /></limits>- <spread-def><numerical-constant unit="s" value="10" /><numerical-constant unit="s" value="10" /><raw-data-def mode="automatic" /></spread-def></qual-parameter-def><numerical-constant unit="s" value="10" /></parameter-def>- <parameter-def name="pip" required="yes"

type="pressure"><raw-data-def mode="automatic" /><numerical-constant unit="s" value="10" /></parameter-def>- <parameter-def name="f" required="yes"

type="frequency"><raw-data-def mode="automatic" /><numerical-constant unit="s" value="30" /></parameter-def>- <parameter-def name="FiO2" required="yes"

type="flow-unit"><raw-data-def mode="automatic" /><numerical-constant unit="s" value="10" /></parameter-def>- <parameter-def name="facecolor" required="yes"

type="facecolor"><raw-data-def mode="manual" /><numerical-constant unit="h" value="1" /></parameter-def></domain></domain-defs><value-defs />- <plans>- <group name="controlled-ventilation">- ...

Impossible to use

for Domain Experts

© Silvia Miksch

© Silvia Miksch © Silvia Miksch

© Silvia Miksch

Visualization Challenges• Hierarchical Decomposition• Conditions• Plan Types• Temporal Order• Compulsory vs. Optional Plans• Cyclical Plans• Temporal Uncertainty

© Silvia Miksch

AsbruView

Interface to thePlan-Representation Language Asbru

Track 1

Track 2

Traffic

© Silvia Miksch

AsbruView’s Dimensions

1. Flow of Time

2. Parallel Plans

Subplan AA

Subplan AB

Plan A

3. S

ubpl

anLe

vels

4. Color

© Silvia Miksch

Anatomy of a Plan

Time

Parallel Plans

Leve

ls

Abort Condition

Suspend Condition

Reactivate Condition

Filter Precondition Complete Condition

Setup Precondition

© Silvia Miksch

AsbruView vs. Flow-Charts

Time

Levels

Parallel Plans

Plan A

Filter-Precondition

Y

N

Find otherplan

Setup-Precondition

Y

N

Abort?

Abortplan

Y

N

N

Fulfillable?

FulfillY

Complete?

PlanCompleted

Y

NSuspend?

dAction

Y

N

Reactivate?

Y

NAbort?

Y

N

© Silvia Miksch

Sequential Plans

Temporal View

Topological View

Time

© Silvia Miksch

Parallel Plans

Topological View

Temporal View

Time

© Silvia Miksch

Some-Any-Order Plans

Topological View

Temporal View

Time

© Silvia Miksch

Cyclical Plans

Topological View

Temporal View

Time

© Silvia Miksch

Level of Detail

Close

Open

© Silvia Miksch

AsbruAsbruAsbruAsbruView

© Silvia Miksch

AsbruAsbruAsbruAsbruView - SopoView

© Silvia Miksch

Evaluation

MetaphorsColorsTwo ViewsTime Annotations

Speed(Plan Placement)

© Silvia Miksch

Asbru’s Usability

• Transfer of Knowledge• Verification and Validation• Evaluation of Protocols• Knowledge Roles• Temporal Dimensions• Reuse• Quality Assurance

© Silvia Miksch

Conclusion

Asbru Clinical Protocols

Asbru not suitable for Physicians

AsbruViewGraphical RepresentationMetaphorsGlyphsTwo Views

© Silvia Miksch

INPUTTime-Oriented

Data ValidationPlan Verification

Patient Data

++

* **Raw Data??

ClinicalProtocols

?Raw Data

Overview: Research Projects

Patient Advocate

V & V

Asgaard/Asbru

VIE-PNN VIE-Nmed

Asgaard/SopoView

VIE-VENTPulsoximetry

SemanticDepth of Field

(SDOF)

© Silvia Miksch←

© Silvia Miksch

VIE-Nmed

←

© Silvia Miksch

VIE-VENT

← © Silvia Miksch

SopoView

←

© Silvia Miksch

Patient Advocate

← © Silvia Miksch

Pulsoximetry

←

© Silvia Miksch

VuV

←

INPUTTime-Oriented

Data ValidationPlan Verification

Patient Data

++

* * *

Raw Data

??

Clinical Protocols

?

Raw Data

© Silvia Miksch

Building BlocksTime-Oriented

Skeletal Planning

INPUTTime-Oriented

Data Validation

InformationVisualization

Temporal Data Abstraction

Plan Verification

Patient Data

++

* * *

Raw Data

??

Clinical Protocols

?

Raw Data

←

© Silvia Miksch

SDOF PrincipleSpatial

Arrangement

2D3D Viewing and

Camera ModelPhotorealisticAdaptive...

Data

Relevanceand Blurring

SelectionDistance...

r ∈ [0; 1] ⇒ b ∈ [1; ∞[ ←© Silvia Miksch

Idea: 2D SDOF Scene

irrelevant

relevant

focus plane

←

© Silvia Miksch

Focus and Context: Threats?

←© Silvia Miksch

Focus, but no Context ...

←

© Silvia Miksch

Focus and Context: Threats?

←© Silvia Miksch

Focus and Context: Covers?

←

© Silvia Miksch

Questions?

http://www.asgaard.tuwien.ac.at/

© Silvia Miksch

Incomes’ Travels Gewerkschaftspolitik: % Erhöhg.KV-Abschlüsse: 1,15%IST-Abschlüsse: 0,58%

innerbetrieb. Erhöhung: normal

TheodorTandler

TechnikerEr hat die HTL besucht.Er hat Berufserfahrung.

Alter: 23 Jahre

00.000

10.000

20.000

30.000

40.000

50.000

60.000

70.000

80.000

90.000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

normal

keine Umrhg.

End-Umrhg.

 +Wechsel

 -Wechsel

Monatliches Einkommenin den Jahren 1-25

FAHRTROUTEN A1: Wechsel zwischen Verwendungsgruppen und Wechsel der Firma

Summe des Einkommens nach ... Jahren

normal keine Umrhg. End-Umrhg.  +Wechsel  -Wechsel Einstufung zu Beginn 5 Jahre 1.619.000 1.619.000 1.619.000 1.619.000 1.619.000 des Arbeitsverhältnisses III 4

10 Jahre 3.604.000 3.530.000 3.530.000 3.640.000 3.531.000 Überzahlung zu Beginn 22%15 Jahre 6.021.000 5.741.000 5.741.000 6.237.000 5.802.000 Umreihung in höhere Verwen-20 Jahre 8.868.000 8.091.000 8.208.000 9.401.000 8.500.000 dungsgruppe geplant nach: III 8 25 Jahre 12.155.000 10.553.000 11.137.000 13.185.000 11.679.000 Max. Zahl VG-Jahre i. d. VG 18

Bruttogehälter (14mal pro Jahr); Rundung auf 1000,-- ÖS; inflationsbereinigt Letzter EB-Sprung i. d. VG bei 16

"normale" Einkommens-entwicklung

... keine Umreihung

in höhere VG

... ohne gepl. Umrhg., aber

Umrhg. am Ende der VG

... mit erfolg-reichem

Fa.wechsel

... mit Fa.wechsel

nach Ar- beitslosigkeit

Annahmen für die "normale"

Einkommensentwicklung

←