task deconstruction and metrics

Robert Sweet, MD

Task Deconstruction and Metrics

Robert M. Sweet, MDAssistant Professor Urologic Surgery

and General SurgeryClinical Director of Simulation

ProgramsFounding Director, CREST

Robert Sweet, MD

Procedure vs. Task vs. Exercise

• Procedure-is a “series of steps taken to accomplish an end”

• Task-“piece of work to be done”• Exercise-”effort performed to develop or

maintain fitness or increase skill”

Webster’s Dictionary

Robert Sweet, MD

Task deconstruction

• Procedures can be broken down into logical components

• Guidelines for the “order by which this occurs”– Safety considerations– Natural progression of skills from simple to complex– Resources and expense

• Independent opportunities for learning loaded on the front end, but sprinkled throughout

Robert Sweet, MD

Cognitive Technical+ Performance=

Dependent

Independent

Automatic

+Communication

Situational Awareness

Closed-Loop Communication

Leadership Transfer

Team Formation and Reformation

Psychomotor

Visual-spatial

Learning Domains in Simulation Training

Robert Sweet, MD

Classify your learners

• Who are your learners• Where are they as far as their level of

understanding?

Robert Sweet, MD

Types of Knowledge

• Declarative– Knowing “what”

• Procedural– Knowing “how”

• Schematic– Knowing “why”

• Strategic– Knowing how and when to apply

Wilson et al, Systems for State Science Assessment, 2005

Robert Sweet, MD

Learning Domains-Cognitive

• “Discriminate the differences between IJ and Subclavian central lines”

• “What would you do if . . . . .”

• “Asking for the proper equipment you need to have available to put in a central line”

• “Name the venous branches leading to the SVC”Bloom’s Taxonomy

Robert Sweet, MD

Robert Sweet, MD

Cognitive Domain Tasks

• Usually good to load on the front-end of the curriculum as it meets the criteria– Safe– Low resources– Logical to understand before doing– Estimated to comprise 75% of what we learn

in procedures (Spencer et al.)

Robert Sweet, MD

Teaching and assessing judgment

Red Llama, Inc.:SimPraxis

Robert Sweet, MD

Communication Domain

• Situational Awareness

• Closed loop communication

• Leadership Transfer• Team Formation and

Reformation

Robert Sweet, MD

Psychomotor/Visual-spatial Domain

Imitation

Manipulation

Precision

Articulation

Naturalization

Dave’s TaxonomyDave, R. H. (1975). Developing and Writing Behavioural Objectives. (R J Armstrong, ed.) Educational Innovators Press

Robert Sweet, MD

Psychomotor/Visual-spatial Domain

Imitation

Manipulation

Precision

Articulation

Naturalization

Dave’s TaxonomyDave, R. H. (1975). Developing and Writing Behavioural Objectives. (R J Armstrong, ed.) Educational Innovators Press

Observing and patterning behavior after someone else

Robert Sweet, MD

Psychomotor/Visual-spatial Domain

Imitation

Manipulation

Precision

Articulation

Naturalization

Dave’s TaxonomyDave, R. H. (1975). Developing and Writing Behavioural Objectives. (R J Armstrong, ed.) Educational Innovators Press

Observing and patterning behavior after someone else

Performance after training, with errors

Robert Sweet, MD

Psychomotor/Visual-spatial Domain

Imitation

Manipulation

Precision

Articulation

Naturalization

Dave’s TaxonomyDave, R. H. (1975). Developing and Writing Behavioural Objectives. (R J Armstrong, ed.) Educational Innovators Press

Observing and patterning behavior after someone else

Performance after training, with errors

Refining technique with few errors

Robert Sweet, MD

Psychomotor/Visual-spatial Domain

Imitation

Manipulation

Precision

Articulation

Naturalization

Dave’s TaxonomyDave, R. H. (1975). Developing and Writing Behavioural Objectives. (R J Armstrong, ed.) Educational Innovators Press

Observing and patterning behavior after someone else

Performance after training, with errors

Refining technique with few errors

Multi-tasking

Robert Sweet, MD

Psychomotor/Visual-spatial Domain

Imitation

Manipulation

Precision

Articulation

Naturalization

Dave’s TaxonomyDave, R. H. (1975). Developing and Writing Behavioural Objectives. (R J Armstrong, ed.) Educational Innovators Press

Observing and patterning behavior after someone else

Performance after training, with errors

Refining technique with few errors

Multi-tasking

High-level of performance on auto-pilot

Robert Sweet, MD

Psychomotor/Visual-Spatial Skills• Basic skills (adapted from Satava et al., 2002)

– Patient positioning– Knowing steps of procedure– Knowing equipment needed– Sterilization procedures– Instrument handling– Bimanual Dexterity– Navigation– Suturing– Knot-tying– Intubation– Palpation– Cannulation– Cutting– Incising– Exploration– Tissue handling

Berkley et al.,2000

Robert Sweet, MD

Psychomotor/Visual-Spatial Skills

• Intermediate Skills (adapted from Satava et al., 2002)– Complex wound closures– Hemorrhage control– Endoscopy– Excision– Anastomosis– Debridement– Complex seldinger

technique

Robert Sweet, MD

Psychomotor/Visual-Spatial Skills

• Advanced skills– Sewing in a “deep hole”– Reconstruction– Adopting techniques to abnormal

anatomical/physiologic situations (cognitive too!) – Advanced minimal invasive skills– New adoptions of technology once done with formal training– Occurring in animal labs

Robert Sweet, MD

With each curricular subtask, choose a learning domain that

predominates

Robert Sweet, MD

Task deconstruction

• Methods to do this?– Look at operative reports/atlases– Practical experience– Remember when you were a student– Ask an introspective student/resident to “keep a diary”

of • What they are learning• Self-assessment of “level” of comprehension• When they are learning it

Robert Sweet, MD

Competence vs. Capability

COMPETENCEWhat individuals know or are able to do in

terms of knowledge, skills, attitudeCAPABILITY

Extent to which individuals can adapt to change, generate new knowledge, and continue to improve their performance

Brit. Med. J. 323:799-803, 2001

Robert Sweet, MD

Learning curves/criterion levels

MASTERY

COMPETENCY

PROFICIENCY

Hypothetical Learning Curve for a technical skill set

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Student AMean

Robert Sweet, MD

Learning curves/criterion levels

MASTERY

COMPETENCY

PROFICIENCY

Hypothetical Learning Curve for a technical skill set

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Student AMean

Simulator Remedial training

Robert Sweet, MD

Learning curves/criterion levels

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Student AMean

MASTERY

COMPETENCYPROFICIENCY

Hypothetical Learning Curve for a technical skill set

Robert Sweet, MD

Metrics for Assessment

• Each task has two components– The performance task itself– A clearly defined scoring scheme/rubric

Robert Sweet, MD

Metrics

Make a list of what is it that is important to measure to determine competence for each subtask?

Establish Criterion levels based on subject matter expert consensus and performance

Robert Sweet, MD

Metrics examples (adapted from Satava et al. 2002)

Quality metrics Errors/”near misses”/events/complications

Blunt end vs. sharp end Missed target Improper delivery of energy Bleeding Final product (leaking or inaccurate) Patient outcomes

Ability to recover from error Efficiency metrics

Time to task Economy of motion Path lengths Response latency Use of resources Use of assistant(s) Proper sequence of steps