2008 © Copyright Incremental Systems Corporation

Using Simulators to Determine How an Expert Operator

Makes Decisions

EPCC Dublin, Ireland

June 14-17, 2009

by Robin Podmore

Incremental Systems, Issaquah, WA

By Understanding how an Expert Operator Makes Decisions we can:

• More accurately assess operator performance.• Develop better training courses• Develop and test better applications• Develop and test better human interfaces

Outline

• What is an Expert?• Simulator Configurations• Operator Decision Making• Summary

ExpertsWhat is an “Expert”?

“Someone widely recognized as a reliable source of technique or skill whose faculty for judging or deciding rightly, justly, or wisely is accorded authority and status by the public or their peers. An expert, more generally is a person with extensive knowledge or ability in a particular area of study.”

Source: Wikipedia, 2008

• Stage 1 Novice – Explicit rules susceptible to context and nuance• Stage 2 Advanced Beginner – Nuance and context begin to be

recognized and incorporated• Stage 3 Competence - Transition from calculated effort to intuitive

solutions• Stage 4 Proficiency – Scenarios are now being recognized as whole

parts, some analysis and conscious choice remains• Stage 5 Expertise – Complete contexts are recognized and

performance is fluid and unselfconscious.

Levels of Expertise Dreyfus and Dreyfus 1986

•Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. New York: Free Press.

Novices vs Experts

Read

Analyze

Explore

Plan

Implement

Verify

Novice Pattern Expert Pattern

PowerSimulator with EPRI OTS Background

• EPRI OTS Power System Model (PSM) developed by Arizona State University / Control Data with EPRI Sponsorship. Dr. Anjan Bose and Dr. Ralph Masiello.

• PACE Power Application Computing Environment- Developed by Incremental Systems and PowerData Corporation.

• PACE and EPRI OTS PSM integrated by IncSys and PowerData with EPRI Sponsorship using EPRI CIM. David Becker and Stephen Lee.

• ESB Ireland Project – First Custom Simulator with Hugh Jones and Dr. Charles Hansen of DSI.

• Emergency Operations with PowerSimulator tutorials developed with SOS International partly based on scenarios developed by EPRI and Quality Training.

Generic PowerSimulator Users• Requirement from 2003 US Northeast Blackout –

Each system operator shall have more than 32 hour of emergency training using simulators.– Midwest ISO (100 operators trained each year) – WECC (200)– PJM (500)– ERCOT (500)– SERC (500)– Florida Reliability Coord. (200)– California Electric Training Advisory Comm (500)– SOS International (600)

• Total - more than 3000 NERC certified operators trained each year.

2008 © Copyright Incremental Systems Corporation

EXTERNAL

CHSAN UXBRDG COPMANOR COPLEY STNTON OAKDLEAMUS

POOL

BEAVERASH

LOCHER

MOSES

JENKINHOMERVEXLEY

WYNHAM

ELLS

FARLIE

GRANGE

DAWSON

TANTONRICTER

KINCAID

DOYLE

200

3*200500600

1200

702*400600

1200

BAKER

CRWFRD

AIRPRT NESTLE

230 KV

138 KV

69 KV

33 KV

80

400

PALCO Map

Generic PowerSimulator PJM Room Setup

PowerSimulator can be used to recruit new engineers and operators

Custom PowerSimulator Users

• North American Users :– Alaskan Railbelt

Systems– Alberta– GCPUD– DCPUD– PG&E– Modesto– Turlock– Burbank– NIPSCO– SIPC– ISO-NE

Custom PowerSimulator Users

• International Projects:– Japan– ONS Brazil– Ireland– Iraq

PG&E CETAC Drills (2009)

2009 Copyright Incremental Systems Corporation

• 5 weeks• 500 total operators• All major utilities in California• Generic PSM for principles• Custom PG&E Model and Northern California Area for Daylong drills• Drill based on report of earthquake off the coast of N. California

Alberta Electric System Operator Restoration Drills

2009 Copyright Incremental Systems Corporation

flash 360

http://www.powersimulator.net/?m=psmCENisku.html

Iraq Ministry of Electricity National Dispatch Center

2009 Copyright Incremental Systems Corporation

• Located Al Ameen, Iraq• New and Incumbent Engineers• Generic PSM for principles• Custom Iraq Model• Remedial Action Schemes

Stations May Have a Combination of Custom Breaker and Generic Breaker Configurations

Travelling Care of US Government

Naturalistic Decision Making

• Typical industry training doesn’t match the way people think. Effective training for critical decision making must be informed by cognitive theories and models.

• Naturalistic Decision Making (NDM) is an area of cognitive science that describes how experienced individuals and teams perform in real-time, mission-critical environments.

• First NDM conference held in 1989.• Ninth NDM conference Covent Gardens June

1990

Example Applications of NDM

• Health Care – Surgeons, Nurses• Command and Control• Aviation• Business and Industrial Applications• Process Control• Material Requirements Planning• Nuclear Power Plant Emergencies• Anesthesiology• Naval Officers in Littoral Environments• Skilled Fighter Pilots

The Challenge of Decision MakingThe Challenge of Decision Making

High Stakes

DynamicSettings

Time Stress Uncertainty

Organizational Factors

Vague Goals

Multiple Players

High Task Loading

Critical Decision Making

NDM Thought Leaders

• Gary Klein – Recognition Primed Decision Model• Mica Endsley – Situation Awareness• Marvin Cohen – Consistent, complete, validated

“Story” with Quick Test• Sallie Gordon – Cognitive Task Analysis

Recognition Primed Decision Model Developed by Gary Klein

Situation

Cues

ActionScript

MentalSimulation

MentalModels

That affect the

That let youcreate

That activate

Generates

Which youAssess by

Using your

Patterns

NDM Applications to Power

• RPDM introduced to Power Industry by Doug Harrington of Team Formation

• RPDM extended by Robin Podmore of IncSys, Frank Greitzer of PNNL and Chuck Johansen and Pam Eye of SOS to reflect:– Need for 100% Error Free Decisions – Quick Test– Distinction between Long and Short Term Memory– Endsley’s Three Levels of Situation Awareness

‹29›

Operator Decision ModelSituation

Cues

That affect the

That let youcreate

That activate

Generates

Which youAssess by

Using your

Which youValidate by

Using your

TimeAvailable?N

MentalSimulation

MentalSimulation

ActionScript

Real World Short termmemory

Long termmemory

Story(Patterns)

MentalModels

MentalModels

Y

Completeness of Story

• Levels of Situation Awareness:– Level I – (What) Operator is monitoring correct

cues– Level II – (So What) Operator has developed a

Story using Mental Simulations and Mental Models to explain meaning of the cues.

– Level III – (Now What) Operator can project what will happen in the future using Mental Simulations and Mental Models.

Patterns

• Using cues to recognize key patterns is a critical step.

• Patterns must be recognized to determine appropriate control actions.

• More experience operators recognize a wider range of patterns and will more quickly detect when a new pattern has emerged.

Examples of Patterns

• System vulnerable to single contingency, line/transformer overloads about to cause cascading thermal outages

• Voltages in an area are weak • Neighboring system is drawing MVARs from our

system• System on verge of voltage collapse• Unit in own control area has tripped• Generation reserves are in-sufficient• Generation is bottlenecked• Load demand is bottlenecked• System has formed multiple islands• Discussion of other patterns ……

‹33›

Mental Models

• An expert power system operator has thousands of relevant mental models in LTM.

• Mental models range along a continuum from simple to complex.

• Required for but not limited to:– Physical, mechanical and electrical characterizations

of different system components and how they work together

– The abilities, sensibilities, limitations and motivations of the system operator’s extended team.

– Operating procedures and policies

PNNL – Human Factors Test LaboratoryIncludes PowerSimulator and Areva DTS

Summary – Current Applications• Operator Decision Model (ODM):

– Explains how an expert power system operator makes decisions.– Used to understand and document the Mental Models and Mental

Simulations used under real-time conditions.– Used to capture knowledge from expert system operators– Used to develop range of scenarios from simple to complex– Used to explain how an expert develops situation awareness.– Used to prepare for NERC System Operator Certification– Used to define gaps in training curricula.– Currently used for over 3000 NERC system operators to maintain

Emergency Training Hours.

Summary Future Applications• The ODM can also be used to:

– Develop better applications– Develop better user interfaces

• and better user interfaces.• The ODM can be used to interview and determine ideal

personality profiles for expert system operators.• On-line tutorials and simulation exercises using ODM can

be used to prepare for NERC certification tests.