Confidence, Error, and Ingenuity in Diagnostic Problem Solving: Clarifying the Role of Exploration and Exploitation

Jenny W. Rudolph Harvard Medical School

Center for Medical Simulation jwrudolph@partners.org

J. Bradley Morrison Brandeis University

International Business School bmorriso@brandeis.edu

Version 1 Winter 2007

DRAFT

Submitted to the Academy of Management for the Annual Meeting in August, 2007

at Philadelphia, PA USA

1

Confidence, Error and Ingenuity in Diagnostic Problem Solving: Clarifying the Role of Exploration and Exploitation

ABSTRACT

This paper examines the role of confidence in how clinicians balance exploring new diagnoses and exploiting known treatments in handling a diagnostic challenge in acute care. Building on in-depth studies of diagnostic problem solving in a time-pressured acute care setting, we develop a formal mathematical model of the dynamic structures that underlie four diagnostic archetypes: fixating, in which a self-reinforcing exploitation structure dominates; diagnostic vagabonding in which a self-reinforcing exploration structure dominates; stalling, in which neither exploration nor exploitation are sustained, and adapting in which structures for both exploration and exploitation are engaged in balance. We clarify the mechanisms that facilitate and prevent switching between and combining exploring diagnoses and exploiting known treatments. This analysis provides both theoretical insights into the interplay of exploration and exploitation and practical suggestions for reducing diagnostic error and enhancing diagnostic effectiveness.

2

In 1999, the US National Academy of Science’s Institute of Medicine rocked the medical world

with its report that medical errors in US hospitals alone may cause as many as 44,000-98,000

deaths a year. While the exact scope of the medical error problem (Hayward & Hofer, 2001) and

how much progress has been made in reducing this rate (Wachter, 2004) is under debate, we do

understand a great deal more about medication errors, hand-off related errors, hospital-acquired

infections, and wrong-side procedures than we did 10 years ago (Campion, 2003; Gandhi, 2005;

Heimann, 2005; Jagsi et al., 2005; Pronovost et al., 2003; Weick, Sutcliffe, & Obstfeld, 2005).

However, one area of medical error remains largely opaque: errors in diagnostic problem

solving. Recent surveys of the literature on diagnostic error as well as early empirical work find

that, although such errors happen at nontrivial rates (2-5% in radiology, pathology, and possibly

in the double digits in other specialties when autopsies are included) they are understudied both

by researchers and by healthcare organizations’ quality tracking efforts (Berner & Graber, 2006;

Graber, 2005; Schiff et al., 2004). Further, these surveys and older research also find that

clinicians themselves are often unaware of some of the diagnostic errors they've made (Elstein,

Shulman, & Sprafka, 1978). Diagnostic errors are also hard to study because they go to the

heart of physician competence; clinicians often feel that such errors are undiscussable

(Christensen, Levinison, & Dunn, 1992; Crosskerry, 2000; Wu, Folkman, McPhee, & Lo, 1991).

Despite the facts that diagnostic errors are often invisible and hard to track, and physicians rarely

receive feedback on the accuracy of their diagnoses, a small number of studies on diagnostic

problem-solving in process shed some on typical patterns of success and failure modes. The

canonical approach to diagnostic problem solving, promoted in medical school and residency, is

the formulation of a differential diagnosis, a list of candidate diagnoses which are then ruled out:

3

Medical problem solving proceeds by selecting a number of diagnostic hypotheses as possible goals, and then testing to see whether one or more of the hypotheses selected can be justified (Elstein et al., 1978, pg. 21).

There is ample evidence, however, in studies of “fixation error”—a process in which a person

sticks with early-developed meanings despite countervailing external cues—that this process

breaks down in healthcare and elsewhere. Instead, clinicians and others often hit upon an early

diagnosis, build confidence in it (or at least don’t lose confidence in it) such that it filters and

even distorts cues the person extracts from the environment (Cook & McDonald, 1988; De

Keyser & Woods, 1990; Elstein et al., 1978; Finkelstein, 2003; Johnson, Moen, & Thompson,

1988; Johnson & Thompson, 1981; Smith & Blankenship, 1991; Voytovich, Rippey, &

Suffredini, 1985; Xiao & MacKenzie, 1995). Rather than updating the diagnostic story based on

reinterpretation of clinical signs and symptoms, a plausible but incorrect diagnosis persists as

confidence in or commitment to that story escalates in a self-reinforcing cycle that stymies

revision (De Keyser et al., 1990; Starbuck, Greve, & Hedberg, 1978; Staw, 1976). Examples of

this from healthcare and elsewhere include the accidental shoot-down in 1988 by the U.S.S.

Vincennes of a commercial Iranian airliner (Cannon-Bowers & Salas, 1998; Roberts &

Dotterway, 1995), the 1994 friendly fire incident in which American pilots killed other

Americans in a Black Hawk helicopter in Iraq (Snook, 2000), the firestorm in Mann Gulch,

Montana in 1949 that killed 13 smoke jumpers (Weick, 1993b), and the death in 2003 of a 17-

year-old girl at Duke University after receiving a heart-lung transplant from an incompatible

donor (Campion, 2003).

Though we know much about fixation error as a type of diagnostic error, it is likely that the

variety and ubiquity of diagnostic errors do not make them amenable to what Reason calls “

‘tokenism’—focusing remedial efforts on preventing the recurrence of specific unsafe acts”

(Reason, 1990, pg 206). Instead, organizational theory can help reduce diagnostic errors and

4

improve patient safety by providing concepts that help us identify generalizable patterns in both

diagnostic error and diagnostic effectiveness. One area of organizational theory that lends itself

to mapping general patterns in diagnostic problem solving is the conceptual dichotomy between

the exploitation and exploration articulated by James March (March, 1991; March, 1996; March,

2006).

Rather than address all diagnostic problem solving, this paper focuses on the problem of

diagnostic error and effectiveness in settings where there is diagnostic ambiguity in an acute care

setting, with some time pressure; settings where the patient can deteriorate over minutes rather

than hours or days such as the operating room (OR), emergency department (ED) or the intensive

care unit (ICU). The diagnostic challenge we study is how a doctor keeps a patient clinically

stable while also confronting a serious, ambiguous clinical problem with the potential to harm

the patient. In such a situation, doctors must explore the clinical terrain, examining clinical signs

and symptoms, and occasionally test or question the soundness of their own diagnostic story,

while also exploiting known treatment algorithms and studies as they pursue therapy to stabilize

the patient (Elstein et al., 1978; Johnson, Hassenbrock, Duran, & Moller, 1982; Klein, Phillips,

Rall, & Peluso, 2006; Klein, Pliske, Crandall, & Woods, 2005). In this situation, doctors must

diagnose an unfamiliar or novel presentation of a problem and, at the same time, pursue routine

treatments or tests quickly.

To model how doctors balance exploration and exploitation to manage this sort of diagnostic

problem, we began by clarifying what we mean by exploration and exploitation. March distilled

central streams of broad organizational literatures in strategy and organizational learning

concerned with when people should pursue what they already know versus try new approaches

into the potent dichotomy between “exploiting old certainties” and “exploring new possibilities”

5

(March, 1991; March, 1996; March, 2006). Exploitation, according to March, includes

approaches like “refinement, choice, production, efficiency, selection, implementation,

execution” (March, 1991: 71) and exploration approaches like “search, variation, risk taking

experimentation, play, flexibility, discovery, innovation” (March, 1991: 71). The concepts of

exploration and exploitation provide a way to examine the processes of inertia and change in an

emerging diagnostic story and doctors’ confidence in how that story fits with external reality.

The expression “exploration” may call to mind search, discovery or even adventure somewhere

“out there.” But the organization studies literature also indicates that exploration in diagnostic

problem solving of any sortin the sense of experimentation, hypothesis testing, and cognitive

flexibilityalso requires the willingness to question one’s own internal expectations and

assumptions (Bartunek, 1984; Kegan, 1994; Nielsen & Bartunek, 1996; Torbert, 1991). Such

internal exploration can be particularly difficult in stressful acute care environments where

psychological and cognitive processes drive people toward exploitation of well-learned

responses (Rudolph & Repenning, 2002; Staw, Sandelands, & Dutton, 1981; Weick, 1993b).

Exploitation, on the other hand, requires having enough confidence in a diagnosis to hold it

firmly in mind and thoroughly pursue relevant treatment algorithms, which in turn provides

evidence about the current clinical problem.

Organizational research concerned with the challenges and trade-offs between exploring new

possibilities and exploiting known certainties has revealed much about when each is optimal

(Carroll, Rudolph, & Hatakenaka, 2002; Gupta, Smith, & Shalley, 2006; March, 1991; March,

1996; March, 2006; Repenning & Sterman, 2002; Torbert, 1991; Weick, Sutcliffe, & Obstfeld,

1999) Through this research we have learned that in situations where certainty is high and

managing a large quantity of routine demands is important, exploiting established knowledge

6

and skills using standard operating procedures, single-loop learning, or managerial controls will

likely be effective (Galbraith, 1972; March, 1991; Rudolph et al., 2002; Sitkin, Sutcliffe, &

Schroeder, 1994; Staw et al., 1981). When certainty is low and the demands on people are novel,

or unfamiliar, exploration of new possibilities through rethinking problems, improvisation,

widening our attention, reframing, and double-loop learning are the best approaches (Cooper &

Gaba, 1989; Galbraith, 1972; March, 1991; Pearson & Clair, 1998; Sitkin et al., 1994; Weick,

1993b, Weick, 1993c #160). We have also learned that the pace and timing of adaptation—how

quickly and when a person or system gives up exploiting existing strategies and explores new

ones or vice versa has an important impact on performance (Ancona & Caldwell, 1992; Cohen,

Freeman, & Thompson, 1998; Gersick, 1988; March, 1991; Sastry, 1997; Waller, 1999).

This previous research tells us what sorts of problems are amenable to tactics of exploration and

which to tactics of exploitation. Computer simulations have provided insight about how the two

activities interact at the organizational level over weeks and months (March, 1991; Sastry, 1997).

We know little, however, about the exact mechanisms by which individuals move back and forth

between exploration and exploitation in short time frames or what triggers or prevents such

movement. To address this gap, we use a grounded theory approach to induce a formal

mathematical model of the interplay of exploration and exploitation in diagnostic problem

solving.

Drawing on theory and data from an in-depth observational study of diagnostic problem solving

by 39 doctors facing the same acute care scenario in high-fidelity simulated operating room

crises (Rudolph, 2003; Rudolph & Raemer, 2004), other studies of diagnostic problem solving,

((Cohen, Freeman, & Wolf, 1996; Cook et al., 1988; Cook & Woods, 1994; De Keyser et al.,

1990; Dörner, 1997; Elstein et al., 1978; Johnson et al., 1982; Xiao et al., 1995), and

7

organizational theory on exploration and exploitation, we develop and analyze a dynamic model

of a single clinician attempting to diagnose and resolve a challenging problem. Through the

model building and simulation process, we are able to induce an internally consistent theory of

how people move back and forth between exploration and exploitation and the impact of this

interplay on diagnostic problem solving. Our analysis provides a new characterization of the

role of confidence in how diagnostic errors come about and how they can be avoided.

METHODS To clarify general patterns of diagnostic error and effectiveness, we developed a mathematical

model of how clinicians move back and forth between or combine exploration and exploitation

in diagnostic problem solving. In contrast to many formal models in the medical decision-

making literature, we did not deduce ours from general principles, but, rather using the methods

of grounded theory, induced it from theory and data found in an in-depth study of diagnostic

problem solving, other studies of diagnostic problem solving, as well as related literature on

exploration and exploitation. While grounded theory is most commonly used to build theory

from raw data using qualitative analysis, the method is not restricted to this application

(Suddaby, 2006). Strauss and Corbin (1994) were proponents of developing formal (or general)

theories grounded in previously generated domain-specific (or substantive) analyses. Strauss and

Corbin (1994) pointed out that Glaser and Strauss (1967) not only advocated using grounded

theory with quantitative (not just qualitative) analysis, but also suggested using it to generate

theory from theory.

Our goal was to develop a general explanation of what facilitates and prevents clinicians moving

back and forth between and combining exploration and exploitation in diagnostic problem

solving, both crucial processes in avoiding diagnostic errors. We chose formal modeling as a

8

tool for enriching theory on diagnostic problem solving for two reasons. First, while the

organizational studies literature is clear about how and when to use exploration and exploitation

separately and has deduced from general principles the idea that exploration and exploitation

must either be combined simultaneously (ambidexterity) or in a pattern resembling punctuated

equilibrium (Gupta et al., 2006), the mechanisms that allow people to or prevent them from

alternating between the two or combining them is much less clear. Since these processes are

hard to study naturalistically, modeling provides a way to synthesize findings from a range of

studies that illuminate the switching process. Second, despite their variety, existing analyses of

exploration and exploitation converge in outlining an inherently dynamic process (Gupta et al.,

2006; March, 1991). Theorizing about such processes without formal models is notoriously

error-prone and can lead to important logical gaps and inconsistencies cf. (Sastry, 1997; Sterman,

1994). Inducing a formal mathematical model from existing data and theory provides an

approach for both identifying structures common to the different narratives and for enforcing the

internal consistency of the emerging theory. Similar formal modeling efforts across a variety of

organizational domains demonstrate how this process clarifies structures common to the different

narratives and enforces the internal consistency of the emerging theory (Black, Carlile, &

Repenning, 2004; Rudolph et al., 2002; Sastry, 1997; Sterman & Wittenberg, 1999). Translating

a narrative theory into a mathematical model loses some of the richness and nuance of the

original. The benefit, however, is an internally and dynamically consistent theory whose central

structures and relationships are explicitly, rather than implicitly, represented.

The genesis of our theory was Rudolph’s in-depth analysis of diagnostic problem solving in

operating room crises (Rudolph, 2003; Rudolph et al., 2004). Following established procedures

for grounded theory building (Strauss et al., 1994; Suddaby, 2006), we used Rudolph’s typology

9

of four diagnostic problem solving modes as our source data and started the theory building

process by translating her text-based constructs and theoretical relationships into the system

dynamics language of stocks, flows, and feedback loops (Forrester, 1961; Sterman, 2000). To

construct the model (which represents our theory) we used a process of constant comparison

between our diagrams and the constructs and relationships identified in other of studies

diagnostic problem solving. (Cohen et al., 1996; Cook et al., 1988; De Keyser et al., 1990;

Elstein et al., 1978; Johnson et al., 1982; Johnson et al., 1988; Klayman, 1988; Klayman & Ha,

1987; Klein et al., 2006; Klein, Orasanu, Calderwood, & Zsambok, 1993; Xiao et al., 1995).

Through this processes of iterative model elaboration and revision, we translated the emerging

set of relationships into a formal mathematical model and then used computer simulation to

analyze it. Lastly, we returned to the Rudolph’s empirical data as well as the literatures on

exploration and exploitation and diagnostic problem solving, noting both similarities and

differences. The result is a theory that addresses gaps in the literature on diagnostic error and in

our understanding of how exploration and exploitation interact: how and when people move

between exploration and exploitation, when they are combined, and the impact on diagnostic

error. 1

FOUR MODES OF DIAGNOSTIC PROBLEM SOLVING: EMPIRCAL DATA

To ground the mathematical model (presented in the next section) in concrete data, we describe

in this section the clinical challenge that served as the basis of our grounded theory building, and

the initial study of that challenge (Rudolph, 2003; Rudolph et al., 2004). The study examined 1 A technical appendix with complete model documentation appears at the end of the paper; this appendix as well as simulation instructions and a simulating version of the model is available from the authors. The model is created and analyzed using the Vensim software, available from Ventanna Systems, see <www.Ventanna.com>.

10

diagnostic problem solving by 39 anesthesia residents in a full-field, high-fidelity medical

simulation that was part of an on-going training program in clinical crisis resource management2

In the simulation scenario studied, the anesthesiologist is called to take over anesthesia in an

operating room where a 29-year-old woman urgently needs an appendectomy. The scenario

presents a common, but serious problem in anesthesia: difficulty with the airway or ventilation.

A variety of diagnoses for the ventilation problem are plausible, but contradictory evidence is

present for each, except one: The patient has exhaled some mucous into the tube, partially

blocking it. Some air can get through the tube, but not enough for the patient to survive. This is

the actual cause of the problem. Treatments addressing problems other than the mucous plug in

the breathing tube will not result in any sustained improvement in the patient’s status. In fact,

the patient’s oxygen saturation level, the amount of oxygen circulating in their blood,

deteriorates steadily and at an increasing rate, unless the doctor identifies the source of the

problem, the partially blocked tube. The patient can have uneven heartbeat and even go into

cardiac arrest if the problem is not rectified. Other personnel in the OR are loosely scripted staff

.

In these settings, described in depth elsewhere (Gaba, 1994; Gaba & DeAnda, 1989), trainees

confront rare and/or difficult medical crisis scenarios in settings equipped and staffed

appropriately for the clinical situation by role-playing simulation staff. The “patient” is a voiced,

computer-controlled plastic mannequin capable of simulating life signs, fluid and electrolyte

balance, airway gas exchange, hemodynamics (e.g., blood pressure, heart rate), and

pharmacological responses to about 100 medications.

2 We use the term “simulation” in two ways in this paper. The first use refers to the initial source data for the study. These data were provided by a full-field, high-fidelity simulation (i.e. the research participant is in a fully equipped, actual OR with a computer controlled mannequin patient). The second use of the term refers to the computer-based simulation we conducted to analyze the behavior of our mathematical model.

11

and faculty of the simulation center: the surgeon, circulating nurse, and scrub nurse. The scenario

ran for about 25 minutes.

Rudolph identified four different problem solving modes for dealing with this diagnostic

challenge, each with different blends of exploration and exploitation. She defined exploitation in

diagnostic problem solving as the degree to which people use existing routines, algorithms, and

heuristics to generate and take specific actions (e.g. therapeutic treatments) to address given

problems within the context of an existing diagnosis or situation assessment. Exploitation in

diagnostic problem solving requires holding a clear diagnostic frame and utilizing known

treatments for that diagnosis. For example, if a doctor who thinks diagnosis A is the problem

uses the first line treatment for diagnosis A and then the second line treatment, she is exploiting

her existing knowledge of the algorithm for addressing Diagnosis A.

Exploration in diagnostic problem solving refers to the breadth of diagnoses and clinical signs

clinicians consider, and the extent to which they test or examine the soundness of these

diagnoses. For example, if a doctor considers 5 possible diagnoses, this is broader exploration

than considering 1. If she carries out diagnostic tests of one or more of these diagnoses, she is

exploring the validity of her current diagnostic story.

To diagnose an acute care problem involving a concrete physical problem (such as a partially

blocked breathing tube), there is a physical, clinical issue to be addressed. In the case of the

partially blocked breathing tube, exploiting a known treatment for the wrong problem, however

skillful, will not resolve the problem. Similarly, exploring new diagnostic possibilities, if not

coupled with determined treatments or tests to stir up feedback, will also not solve the problem.

Conceptualizing exploration and exploitation as two orthogonal dimensions of sensemaking

allowed these modes to be mapped as a function of different levels of each (see Figure 1).

12

In the first mode, diagnostic problem solving that involves both little exploration and

exploitation, Stalled problem solvers gain little traction on any problem. Not surveying much of

the external terrain and at the same time not exploiting existing knowledge or rules of thumb,

these doctors found it difficult to generate and build confidence in plausible stories about what

was happening that would help them take action; they were stalled in moving toward a resolution

of the diagnostic challenge. In a second mode, some doctors struck on a plausible story early in

their efforts, quickly built confidence in it, and, finding it adequate did not explore either the

external terrain or their own sensemaking itself much more. This allowed them to exploit

standard operating procedures for addressing their view of the problem. They exploited this

plausible story as a lens that focused how they perceived events and did not update it in a way

that improved its fit with external reality. This left them Fixated on a story that did not resolve

the diagnostic challenge. In a mirror image of this fixated sensemaking mode, exploring the

external terrain extensively without exploiting existing knowledge, tests, or standard operating

Four Diagnostic Modes

Exploration Low High

Exploitation H

igh Low

Fixating Adapting

Stalling Diagnostic

Vagabondinng

13

procedures created a sensemaking approach that jumped, like a vagabond moving from place to

place, from plausible story to plausible story3

MODEL STRUCTURE

without taking action that provided feedback on

the accuracy of any one. Without clear feedback, these Diagnostic Vagabonds did not take action

that resolved the diagnostic challenge. Lastly, those in the Adaptive sensemaking mode explored

the external terrain (surveying different aspects of it) and explored the internal terrain by testing

their diagnostic story as they went though studies and tests. In this mode clinicians also exploited

standard operating procedures that generated concrete feedback to help update a plausible story

(or stories) about the crisis in a way that improved its accuracy enough to resolve the crisis.

OVERVIEW Our model assumes that the doctor observes some problem with the patient, and this prompts her

to generate diagnoses about the cause of the problem often in an automatic, unconscious process

so that she can take action. One of these diagnoses is the leading one in her mind. In our model

call this leading diagnosis the current diagnosis. The current diagnosis serves a dual role. First,

the current diagnosis may be a catalyst for action, guiding therapeutic interventions that might

improve the patient’s condition if the diagnosis is correct. Second, the current diagnosis in some

cases may be treated as a hypothesis that can be held up for testing by experiments to gather data

that shape the current diagnostic story. Since the goal of our model is to uncover patterns in how

clinicians move back and forth between or combine exploration and exploitation, we streamline

the challenge facing the clinician to set these patterns in stark relief. Thus, the challenge for the

stylized diagnostician in our model is to 1) hit upon the diagnosis that solves the problem (e.g. as

with our source data, only one diagnosis will resolve the clinical issue); and 2) to determine that

3 We've adapted Dietrich Dörner's term "thematic vagabonding" which he uses to describe a similar phenomenon among public officials conducting scenario planning simulations. Dörner, D. 1997. The Logic of Failure: Recognizing and avoiding error in complex situations. New York: Perseus.

14

it is indeed the diagnosis that will work. Therefore, in our model we assume that once the

diagnostician has determined the right diagnosis, proper treatment and improved patient status

will follow.

Linking to Rudolph’s four problem solving modes, Stalling occurs when the diagnosticians do

not generate a current diagnosis. Fixating occurs when they do not discard their leading or

current diagnosis and never discover the one that solves the problem. Vagabonding occurs when

they jump from current diagnosis to current diagnosis, perhaps hitting upon the one that solves

the problem, but not recognizing that it will resolve the problem and therefore discarding it.

Adaptive problem solving occurs only when the clinicians both discover the correct diagnosis

and determine that it is the correct one.

CONFIDENCE IN CURRENT DIAGNOSIS The central construct in our model is Confidence in the Current Diagnosis (See Figure 2.) We

focus our analysis on this variable because previous research suggests it is the fulcrum on which

exploration and exploitation is balanced. Studies of tactical decision-making under stress

(Cannon-Bowers et al., 1998), problem detection (Klein et al., 2005; Mandler, 1982), and

naturalistic decision-making in the field (Klein et al., 1993; Snook, 2000; Zsambok & Klein,

1997) indicate that gaining and losing confidence in a particular diagnosis or situation

assessment influences whether the problem solver sticks with the current assessment and exploits

treatment algorithms for that problem or steps back to reconsider and explore new possibilities.

We operationalize Confidence in Current Diagnosis as a measure ranging from 0 to 1 to

represent of the diagnostician’s current belief about the likelihood that the current diagnosis is

the correct one. When the first diagnosis emerges, the diagnostician has a beginning estimate

15

given by the Confidence in a New Diagnosis. We will choose different values for this initial

confidence level to model the different problem solving modes.

Figure 2: Confirming the Current Diagnosis

As the doctor takes actions, observes results, and monitors the patient’s progress, the Confidence

in the Current Diagnosis may go up or down. The primary means for the doctor to update the

Confidence in Current Diagnosis is to conduct Treatments and Studies that discover evidence

regarding fit of the diagnosis with external signs and symptoms. Figure 2 shows that as

Confidence in the Current Diagnosis goes up, the doctor conducts more Treatments and Studies.

This in turn increases the rate of Discovering Evidence. The evidence the doctor discovers may

cause him to gain or lose confidence in the current diagnosis. In the Confirmation loop depicted

in Figure 2, discovering more evidence leads to Gaining Confidence and thus even greater

Confidence in the Current Diagnosis. Continuing to discover evidence supportive of the current

diagnosis, which in turn increases confidence in the diagnosis, is a reinforcing process.

Confidence inCurrent Diagnosis

Treatmentsand Studies

Effect ofConfidence

RDiscovering

Evidence+

Confidence in aNew Diagnosis

+

+

+

Confirmation

GainingConfidence in

Diagnosis+

+

16

Alternatively, treatments and studies may lead to discovering evidence that may disconfirm the

current diagnosis. In this scenario, the discovery process leads to Losing Confidence in the

Current Diagnosis, as shown in Figure 3.

Figure 3: Disconfirming the Current Diagnosis

Conducting treatments and studies in diagnostic problem solving generally involves following a

reasonably standardized algorithm. As more treatments and studies are completed, the discovery

of evidence reduces the remaining quantity of Undiscovered Evidence. Figure 4 shows the

balancing loop formed by these limits. Figure 4 also adds two constructs representing

parameters that we will vary in the simulations we show in the next section. Strength of

Confidence Effect describes the degree to which increases in confidence trigger more intense

treatments and studies relative to the baseline rate given by the Propensity to Treat and Study.

These two parameters may vary across the various problem solving modes (Stalled, Fixated,

Diagnostic Vagabonding, and Adaptive.)

Confidence inCurrent Diagnosis

Treatmentsand Studies

Effect ofConfidence

R

Disconfirmation

DiscoveringEvidence

+

Confidence in aNew Diagnosis

+

+

+

LosingConfidence in

Diagnosis+

-

B

Confirmation

GainingConfidence in

Diagnosis+

+

17

Figure 4: Pursuing the Treatment Algorithm

In addition to conducting treatments and studies, another way the doctor increases or decreases

confidence in the current diagnosis is to observe the evolution of the patient’s status. In

Rudolph's study of diagnostic problem solving on which we draw, the patient status deteriorated

steadily until a correct diagnosis was found and a proper intervention was administered. Our

model captures the impact of this external situation on the doctor’s internal confidence in their

diagnosis implicitly: in the absence of any treatments and studies, confidence in the diagnosis

declines over time towards zero.

Following data from a range of studies on diagnostic problem solving cited earlier, in our model,

decreases in confidence in the current diagnosis trigger a search for alternative possibilities.

Figure 5 shows a reinforcing loop describing this search process.

Figure 5: Proposing New Diagnoses

Confidence inCurrent Diagnosis

Treatmentsand Studies

Effect ofConfidence

R

UndiscoveredEvidence

-

+

Limits toEvidence

+ B

Disconfirmation

DiscoveringEvidence

+

Propensity toTreat and Study

+

Strength ofConfidence

Effect

TreatmentAlgorithm

Confidence in aNew Diagnosis

+

+

+

+

LosingConfidence in

Diagnosis+

-

B

Confirmation

GainingConfidence in

Diagnosis+

+

18

So far, the model represents the processes of generating alternatives and testing them by

pursuing treatments and studies to generate evidence. The model currently shows that doctors

must allocate effort between generating new diagnoses (the Proposing Diagnoses loop) in an

exploratory process and treating the currently held diagnosis in a process that exploits known

treatment algorithms (in the Treatment Algorithm loop). Figure 6 adds one additional challenge

that arises as doctors attempt to interpret cues from the external clinical environment. Evidence

from studies of problem detection and fixation error indicate that as Confidence in the Current

Diagnosis rises, the Willingness to Question the Diagnosis goes down, at least until some critical

threshold of discrepancies is reached (Klein et al., 2005; Louis & Sutton, 1991; Mandler, 1984).

With less willingness to question the diagnosis, the doctor places less Weight on Accuracy,

which is the term we give to how heavily the doctor considers evidence from the outside world.

With less Weight on Accuracy, Losing Confidence in the Current Diagnosis is less than it would

otherwise be. (An analogous link from Weight on Accuracy to Gaining Confidence in Diagnosis

with a positive polarity is omitted from the diagram for simplicity.) Consequently, confidence in

the current diagnosis does not decrease. The result is a reinforcing process, highlighted in red in

Confidence inCurrent Diagnosis

Treatmentsand Studies

-Search

Activiites

Generating New andAbandoning Old

Diagnoses-

+

Effect ofConfidence

R

R

UndiscoveredEvidence

-

+

Limits toEvidence

+ B

Disconfirmation

ProposingDiagnoses

DiscoveringEvidence

+

Propensity toTreat and Study

+

Strength ofConfidence

Effect

TreatmentAlgorithm

Confidence in aNew Diagnosis

+

+

+

+

LosingConfidence in

Diagnosis+

-

B

Confirmation

GainingConfidence in

Diagnosis+

+

19

Figure 6. This loop is an important addition. Taken alone, this loop would lead to locking in on

a diagnosis that does not address the patient's problem. When the other loops are also active,

they can work to prevent this loop from dominating the behavior of the system.

Figure 6: Interpreting the Data

RESULTS We developed a formal mathematical model consistent with the causal loop structure described

in the preceding section. This section presents results from simulating the behavior of this

system dynamics model under various scenarios. Our objective was to use simulation analysis

to clarify patterns of switching or failing to switch between exploratory and exploitative

Confidence inCurrent Diagnosis

Treatmentsand Studies

-Search

Activiites

Generating New andAbandoning Old

Diagnoses-

+

Effect ofConfidence

R

R

Willingness toQuestionDiagnosis-

UndiscoveredEvidence

-

+

Limits toEvidence

+

R

B

Disconfirmation

ProposingDiagnoses

DiscoveringEvidence

+

Propensity toTreat and Study

+

Strength ofConfidence

Effect

TreatmentAlgorithm

Confidence in aNew Diagnosis

+

Interpreting theData

Weight onAccuracy

+

+

+

+

LosingConfidence in

Diagnosis+

-

B

Confirmation

+

GainingConfidence in

Diagnosis+

+

20

activities that produce the problem solving modes observed in the field study from which we

induced our theory.

Recall that the central construct in our model is the Confidence in Current Diagnosis. An

adaptive problem solving process will generate declining confidence in a diagnosis that is

incorrect eventually leading to replacing the flawed diagnosis with a new one. If the next

diagnosis is also incorrect, the process should again deplete confidence and yield a replacement,

and so on until the current diagnosis is correct. When the current diagnosis is the correct one, the

process should exhibit increasing confidence and a steady-state convergence on the correct

diagnosis. The replacement of one diagnosis with another is modeled as a discontinuous process

triggered when the confidence in the current diagnosis drops below a specified threshold value.

In the model, this trigger instantly replaces the current diagnosis with a new one for which the

value of confidence is given by the Confidence in New Diagnosis. We also control for any

effects of random search by assuming that all diagnosticians generate alternative diagnoses in the

same sequence. Specifically, we say that the first, second, and third diagnoses considered are

incorrect, the fourth is correct, and the fifth and all others after that are incorrect. In the

simulation results that follow, the time scale begins with the first diagnosis in place at a

confidence set by the value of the Confidence in New Diagnosis parameter. (It is trivial in this

model to replicate the stalled mode of problem solving, in which no diagnosis is generated, so we

will not comment further on this mode.) Table 1 shows the parameters used to generate the

simulated problem solving modes in the following graphs.

21

Table 1: Summary of Parameter Settings in Diagnostic Problem Solving Scenarios

Confidence in New

Diagnosis

Propensity to Treat

and Study

Strength of Confidence

Effect

Rationale

Units Fraction 1/minute Dimensionless Fixating 0.75 1 1 Overconfident in

proposed diagnosis Diagnostic Vagabonding

0.5 0.3 3 Cautious to take action

Adapting 0.5 1 1 Willing to question and to act

Fixating The left panel of Figure 7 shows the trajectory of Confidence in Current Diagnosis over time in

our first simulation. The parameter for Confidence in New Diagnosis is set so the diagnostician

begins with a relatively high level of confidence. The first diagnosis the diagnostician generates

is incorrect (by assumption and consistent with the field study data). Based on this first

diagnosis, the doctor takes some first steps in the treatment and study algorithm. Confidence

begins to decline somewhat as the Disconfirmation loop acts to reduce confidence. As shown in

the right panel of Figure 7, Weight on Accuracy begins to climb as confidence falls. However,

the diagnostician’s strong confidence in the diagnosis keeps the weight on accuracy low. The

diagnostician continues experiencing the situation and creating meaning that supports the current

diagnosis, so eventually the decline in confidence is halted. Confidence reaches its lowest point,

crossing a tipping point, and then as confidence begins to climb, the Weight on Accuracy begins

to fall. The process of reconstructing meaning to support the current diagnosis gains strength.

The Interpreting the Data loop reinforces the current diagnosis, and because the loop is so strong,

the diagnostician does not move on to any other diagnosis. The combination of a high initial

confidence in a diagnosis and the strong reinforcing effects of the Interpreting the Data loop

22

result in fixating, a pattern of diagnostic problem solving in which the doctor’s confidence grows

above a tipping point threshold and is then propelled toward a steady-state convergence in which

the doctor is completely confident in the incorrect diagnosis. This mode of problem solving fails

because overconfidence in the current diagnosis suppresses the available disconfirming evidence,

so the current diagnosis locks in prematurely, squeezing out the search for alternative diagnoses,

and the doctor never has a chance to find the correct diagnosis.

Figure 7

Diagnostic Vagabonding Some instances of problem solving do not begin with such high confidence in new diagnoses.

Our next simulation models a process that starts at a more moderate level of confidence when a

new possible diagnosis is generated. The stylized doctor in this simulation not only has less

confidence in new diagnoses but also has a slower approach to conducting treatments and

studies. We model this slower approach with a lower parameter for the baseline Propensity to

Treat and Study and a higher setting for the Strength of Confidence Effect. When confidence is

high (low), the rate of conducting treatments and studies will be higher (lower), and this latter

parameter moderates how influential Confidence in Current Diagnosis is in adjusting the rate of

Confidence in Current Diagnosis1

0.85

0.7

0.55

0.40 4 8 12 16 20 24 28 32 36 40

Time (Minute)

Weight on Accuracy0.4

0.3

0.2

0.1

00 4 8 12 16 20 24 28 32 36 40

Time (Minute)

23

conducting treatments and studies. The stylized doctor in this simulation is quite capable of

generating new diagnoses but is neither particularly sure of them nor prone to pursue treatments

and studies based on them.

The left panel of Figure 8 shows the trajectory of Confidence in Current Diagnosis over time in

our second simulation. The first diagnosis, which is once again an incorrect one, emerges with

confidence at a moderate level. Just as in the previous simulation, confidence begins to decline

as the Disconfirmation loop acts to reduce confidence, and Weight on Accuracy begins to

increase. But, whereas in our previous simulation high confidence led to interpreting experience

in a way that supported the diagnosis, now with more moderate levels of confidence, the Weight

on Accuracy continues to increase. With increasingly accurate interpretations, confidence

continues its decline and drops below the critical threshold that triggers switching to a new

diagnosis. The doctor properly rejects the first (incorrect) diagnosis. Diagnosis number two

replaces it as the leading diagnosis, and confidence resets to the level of Confidence in New

Diagnosis. Diagnosis number two is another incorrect diagnosis, so confidence falls, eventually

dropping below the critical threshold, and the diagnosis is rejected and replaced. Diagnosis

number three enters and is similarly rejected.

Diagnosis number four becomes the current diagnosis, but now the pattern is different.

Diagnosis number four, unbeknownst to the clinician initially, is the correct diagnosis.

Nevertheless, the early trajectory of confidence is quite similar to that for the previous diagnoses.

The first small amount of treatment and study still does not provide convincing evidence that it is

the correct diagnosis, so confidence declines. The Effect of Confidence as it declines is to

suppress the doctor’s conduct of treatments and studies, so the doctor misses the opportunity to

24

discover evidence to support this correct diagnosis. Confidence continues to fizzle, albeit

somewhat more slowly in this case because there is some supporting evidence, and drops below

the critical threshold. The correct diagnosis number three is rejected, and diagnosis number four

enters as the current diagnosis. Now with only incorrect diagnoses to consider, the pattern of

new diagnosis and eventual rejection continues. In this simulation, the reinforcing Confirmation

loop works as a vicious cycle, squeezing out the conduct of treatments and studies that would

allow the doctor to discover enough evidence to confirm the correct diagnosis. The result is

vagabonding, a pattern of diagnostic problem solving in which the doctor’s confidence in each

new diagnosis declines below the critical threshold even for the correct diagnosis, so the doctor

gets stuck in a steady state of finding new alternative diagnoses but not discovering enough

about these diagnoses to reach the proper conclusion. This mode of diagnostic problem solving

fails because the low levels of confidence in diagnoses suppress the rate of treatments and

studies so much so that even the correct diagnosis is rejected.

Figure 8

Confidence in Current Diagnosis0.6

0.45

0.3

0.15

00 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45

Time (Minute)

Weight on Accuracy1

0.85

0.7

0.55

0.40 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45

Time (Minute)

25

Adapting Some instances of diagnostic problem solving occasion more rapid and complete conduct of a

treatment and study algorithms. Our next simulation models a process in which the doctor

readily generates new diagnoses but also confidently takes action in the empirical world to

conduct treatments and studies in order to evaluate the veracity of a proposed diagnosis. This

scenario differs from the previous one in the parameters that moderate the strength of the

Confirmation loop. The stylized doctor in this scenario has a higher Propensity to Treat and

Study and a lower Strength of Confidence Effect, which means that low levels of confidence

have less effect to suppress action in the realm of treatments and studies. The left panel of

Figure 9 shows the trajectory of Confidence in Current Diagnosis over time under those

conditions.

Figure 9

.

The pattern begins exactly as it did in the vagabonding mode with a moderate degree of

confidence that declines The diagnostician in this scenario holds onto the diagnosis longer than

in the vagabonding case because here the diagnostician conducts more treatments and studies.

The evidence discovered is interpreted as partially supportive of the current diagnosis, at least to

Confidence in Current Diagnosis1

0.75

0.5

0.25

00 4 8 12 16 20 24 28 32 36 40

Time (Minute)

Weight on Accuracy1

0.75

0.5

0.25

00 4 8 12 16 20 24 28 32 36 40

Time (Minute)

26

the extent that continuing the treatment and study algorithm is still deemed worthy. Each

diagnosis is held up for a more thorough examination to determine its merits. Nevertheless, the

first diagnosis is incorrect, so confidence eventually drops below the critical threshold for

rejection and replacement. The pattern continues with a thorough examination and subsequent

rejection of diagnoses two and three. When the diagnostician begins to consider diagnosis

number four, the correct one, confidence initially begins to decline as before. However, the rate

of conducting treatments and studies is high enough to discover evidence that begins to build

confidence in the current diagnosis. Increasing confidence in this diagnosis increases that rate of

doing treatments and studies, discovering more evidence, bolstering confidence even more.

Moreover, increasing confidence boosts degree of favorable interpretation as the Interpreting the

Data loop acts to reduce the Weight on Accuracy. The diagnostician pursues the treatment and

study algorithm to completion and converges on a steady state choice of the correct diagnosis.

Figure 10 shows the degree of Treatments and Studies Completed for both this mode and the

previous mode of vagabonding. A distinguishing feature of this mode of problem solving is that

the doctor accomplishes a much greater portion of the treatment and study algorithm before

moving on to the next diagnosis.

Figure 10

27

The result is adapting, a pattern of diagnostic problem solving in which the doctor’s confidence

is high enough to allow for a more complete examination of each hypothesis before the rejection

of incorrect diagnoses and that enables the doctor to discover and confirm the correct diagnosis.

This mode of problem solving succeeds because the doctor strikes an appropriate balance

between considering new diagnoses and sticking with them long enough to gather evidence to

evaluate them. The adaptive diagnostic problem solver is able both to hold up the diagnoses for

examination and to take action to conduct the examination.

The simulation results in this section show how the feedback structure in our model of diagnostic

decision making (Figure 6) generates the modes of diagnostic problem solving observed in the

field study. Stalling occurs when the diagnosticians do not generate proposed diagnoses.

Fixating occurs when they do not discard an incorrect diagnosis, never discovering the correct

hypothesis. Vagabonding occurs when they do not determine a correct diagnosis to be so and

thus prematurely discard it. Adaptive problem solving occurs only when they both discover the

correct diagnosis and determine that it is the correct one.

Treatments and Studies Completed4

3

2

1

00 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45

Time (Minute)

Treatments and Studies Completed : vagabonding stepsTreatments and Studies Completed : adaptive problem solving steps

28

DISCUSSION Our study offers three contributions to understanding how to avoid diagnostic error and move

toward effective diagnostic problem solving. First, we offer a clear picture of the dynamic

structures that produce four general modes in diagnostic problem solving. Second, we clarify the

mechanisms that make exploratory and exploitative diagnostic activities self- reinforcing and the

mechanisms that allow people to switch between them and combine them. Third, whereas the

liability of exploitation has been well understood via studies and fixation error, the liability of

exploration in the diagnostic context has been left obscure. We clarify this liability by describing

how excessive exploration and inadequate exploitation are linked.

Success and Failure Modes in Diagnostic Problem Solving Diagnostic problem solving in time-pressured, acute care settings is not a situation where

“meaning already exists and is waiting to be found.” Rather it “awaits construction that might

not happen or might go awry…” (Weick, 1995a: 15). In the fixating problem-solving mode,

doctors’ initially high level of confidence in their diagnosis is a launching pad for exploiting

known treatment algorithms via the treatment algorithm loop. The “disconfirmation,” loop loses

strength as confidence in their current diagnosis pushes a self-reinforcing process. This process

finds those in the fixating mode interpreting the feedback they got as yet further confirmation of

their current story in the “interpreting the data loop”. Exploiting the current diagnosis heavily

without exploring other aspects of the terrain, or questioning their own diagnosis suppressed the

updating processes that might have improved the accuracy of the story–its fit with external

reality. In the fixating mode, the Interpreting the Data loop dominates the system behavior,

locking them into a mode of excessive exploitation and preventing the doctors from exploring

alternatives that might steer them to an effective diagnosis.

29

Studies of fixation error (also known as premature closure or tunnel vision), generally conclude

that broadening the range of alternatives considered is the needed antidote. Surprisingly, the

example of the diagnostic vagabonds showed that exploring a broader range of possibilities is not

enough on its own to allow problem solvers to identify and treat the patient’s problem. In

contrast to the stalled and fixated sensemaking modes, the diagnostic vagabonds generated a

wide range of plausible diagnoses. Though they readily shifted among diagnostic stories, they

were not as readily able to assess the accuracy of these stories. Failing to exploit treatments or

studies, the diagnostic vagabonds lacked adequate feedback that would allow them to improve

the fit between of their diagnostic stories and the external environment. Their repeated hesitation

to take action that would generate useful feedback allowed the Disconfirmation loop to

overwhelm the system, leading to frequent introduction of new alternatives by the Proposing

Diagnoses loop. They got locked in to a mode of excessive exploration.

These failure modes suggest that intertwining exploration and exploitation may enhance and

bolster the effectiveness of each activity. In the adaptive sensemaking mode, the stylized doctor

launched into action with a higher Propensity to Treat and Study and a lower Strength of

Confidence Effect. The result was that low levels of confidence had a less suppressive effect on

action in the realm of treatments and studies than it did on the Diagnostic Vagabonds. Thus,

when exploiting known approaches for a given diagnostic story, they proceeded relatively

unencumbered by declining confidence in the diagnosis. Failure was still a trip-wire for

exploring other diagnoses, but with more complete information, the doctor was able to recognize

a correct diagnosis when she found one. Conducting treatments and studies in the “treatment

algorithm loop” combined with exploration in the “proposing diagnoses loop” to yield a mode of

balanced diagnostic problem solving characterized by effective switching between exploitation

30

and exploration. Accomplishing this balance involves the willingness to question one’s current

diagnosis and translating that questioning toward heightening interest in external evidence . The

combination of exploitation (more treatments and studies) and exploration (more alternative

diagnoses) based on feedback allowed those in the adaptive sensemaking mode to assess the

accuracy of plausible diagnostic stories, rule some out, and resolve the underlying problem.

Mechanisms for switching (or not) between exploration and exploitation. Existing research and theory suggests that combining exploration and exploitation is difficult for

an individual or small group under resource constraints, time pressure and possible threat of a

crisis (Gupta et al., 2006; Rudolph et al., 2002; Staw et al., 1981). Time-constrained, resource-

scarce acute care situation situations, this research suggests, are likely to produce exploration or

exploitation, since it is difficult to switch between exploratory versus exploitative mindsets, each

activity tends to be self-reinforcing, and both compete for scarce resources (Gupta et al., 2006;

March, 1991; March, 1996; Rudolph et al., 2002; Staw et al., 1981). Our data help “unpack”

how each of these processes comes about. The exploitative confirmation loop coupled with a

high level of confidence seems to create a mind-set and reinforcing process that captures fixated

problem solvers attentional capacity. The exploratory disconfirmation loop coupled with lower

level of confidence appears to create a mind-set and reinforcing process that hoards diagnostic

vagabonds attentional capacity.

In contrast, in adaptive sensemaking, however, exploration and exploitation appear to be

combined. The dynamics of diagnostic problem highlighted in our model, and missing from

earlier discussions of exploration and exploitation facilitate the combining of exploitation and

exploitation. The first loop crucial to this process is exploring one’s diagnosis by questioning it

(the interpreting data loop). Skeptically reviewing one’s own diagnosis in the interpreting data

31

loop is the linchpin that connects exploration and exploitation, which otherwise tend toward

separate, competing paths. This loop captures the types of activities that have been described as

“having your simplifications but doubting them” (Weick et al., 1999), deutero-learning (Bateson,

1972), double-loop learning (Argyris, Putnam, & Smith, 1985; Torbert & Associates, 2004),

premise reflection (Schwandt, 2005), and mindfulness (Langer, 1989). These are all forms of a

meta-level of awareness that link exploratory to exploitative activities. In our model, it was the

link between the exploitative Treatment Algorithm loop and the exploratory Interpreting Data

loop that made it possible for adaptive problem-solvers to move toward an effective solution.

This process protects problem solvers from the powerful forces that can tip them into either

exploration-only or exploitation-only traps.

Failure Traps Revisited

The literature on exploration and exploitation has highlighted the fact that exploratory activities

can be self reinforcing self-reinforcing because returns on exploration take longer, are often

uncertain, and fraught with failure which prompts more search for yet other new ideas to mitigate

the failure. These characteristics have lead scholars to characterize recursive search activities as

leading to ‘failure traps’. In the terms of our model, these exploratory activities are self

reinforcing because the diagnoses generated in the proposing diagnoses loop are incorrect or it

Our data suggest another explanation for why exploration alone and self reinforcing. A liability

of exploration revealed by our model, is not related to whether the diagnosis is correct or

incorrect, but rather, the liability of vagabonding occurs because a reinforcing feedback

structure, the Confirmation loop, acts as a vicious cycle so that continuous exploration of new

alternatives squeezes out the exploitative activities of the Treatment Algorithm loop to act on

these continuously changing ideas.

32

33

REFERENCES Ancona, D. G., & Caldwell, D. F. 1992. Bridging the boundary: External activity and performance in

organizational teams. Administrative Science Quarterly, 37: 634-665.

Argyris, C., Putnam, R., & Smith, D. M. 1985. Action Science: Concepts, Methods and Skills for Research and Intervention. San Francisco: Jossey-Bass.

Bartunek, J. M. 1984. Changing interpretive schemes and organizational restructuring: The example of a religious order. Administrative Science Quarterly, 29: 355-372.

Bateson, G. 1972. Steps to an ecology of mind. New York: Ballantine Books.

Berner, E. S., & Graber, M. L. 2006. Overconfidence as a Cause of Diagnostic Error in Medicine. Birmingham, AL: Department Of Health Services Administration, University of Alabama at Birmingham.

Black, L., Carlile, P., & Repenning, N. P. 2004. A dynamic theory of expertise and occupational boundaries in new technology implementation: Building on Barley's study of CT scanning Administrative Science Quarterly, 49(4): 572-607.

Campion, E. W. 2003. A death at Duke. New England Journal of Medicine, 348(12): 1083-1084.

Cannon-Bowers, J. A., & Salas, E. 1998. Making Decisions under Stress: Implications for Individual and Team Training. Washington, DC: American Psychological Association.

Carroll, J. S., Rudolph, J. W., & Hatakenaka, S. 2002. Learning from experience in high-hazard industries. Research in Organizational Behavior, 24: 87-137.

Christensen, J. F., Levinison, W., & Dunn, P. M. 1992. The heart of darkness: the impact of perceived mistakes on physicians. Journal of General Internal Medicine, 7: 424-431.

Cohen, M. S., Freeman, J. T., & Thompson, B. 1998. Critical thinking skills in tactical decision making: a model and a training strategy. In J. Cannon-Bowers, & E. Salas (Eds.), Decision Making Under Stress: Implications for Individual and Team Training. Washington, D.C.: American Psychological Association.

Cohen, M. S., Freeman, J. T., & Wolf, S. 1996. Meta-recognition in time stressed decision making: Recognizing, critiquing, and correcting. Human Factors, 38(2): 206-219.

Cook, R. I., & McDonald, J. S. 1988. Cognitive tunnel vision in the operating room: Analysis of cases using a frame model. Anesthesiology, 69: A-497.

Cook, R. I., & Woods, D. 1994. Operating at the sharp end: The complexity of human error. In B. S. Bogner (Ed.), Human Error in Medicine: 255-310. Hillsdale, NJ: Lawrence Earlbaum Associates.

Cooper, J., & Gaba, D. M. 1989. A strategy for preventing anesthesia accidents. International Anesthesiology Clinics, 27(3): 148-152.

Crosskerry, P. 2000. The feedback sanction. Academic Emergency Medicine, Seven: 1232-1238.

De Keyser, V., & Woods, D. D. 1990. Fixation errors: Failures to revise situation assessment in dynamic and risky systems. In A. G. Colombo, & A. Saiz de Bustamante (Eds.), Systems Reliability Assessment: 231-251. Amsterdam: Kluwer.

Dörner, D. 1997. The Logic of Failure: Recognizing and avoiding error in complex situations. New York: Perseus.

34

Elstein, A. S., Shulman, L. S., & Sprafka, S. A. 1978. Medical Problem Solving: An Analysis of Clinical Reasoning. Cambridge, MA: Harvard University Press.

Finkelstein, S. 2003. Why Smart Executive Fail: And What You Can Learn from Their Mistakes. New York: Portfolio.

Forrester, J. W. 1961. Industrial Dynamics. Portland, OR: Productivity.

Gaba, D. 1994. Human work environment and simulators. In R. D. Miller (Ed.), Anesthesia: 2635-2679. New York: Churchill Livingstone.

Gaba, D. M., & DeAnda, A. 1989. The response of anesthesia trainees to simulated critical incidents. Anesth Analg, 68(4): 444-451.

Galbraith, J. R. 1972. Organization design: An information processing view. In J. W. Lorsch, & P. R. Lawrence (Eds.), Organization Planning: Cases and Concepts: 49-74. Homewood, IL: Richard D. Irwin, Inc.

Gandhi, T. K. 2005. Fumbled handoffs: one dropped ball after another. Annals of Internal Medicine, 142(5): 352-358.

Gersick, C. 1988.

Glaser, B. G., & Strauss, A. L. 1967. The discovery of grounded theory: Strategies for qualitative research. New York: Aldine.

Graber, M. 2005. Diagnostic errors in medicine: A case of neglect. Journal on Quality and Patient Safety, 31(2): 112-119.

Gupta, A. K., Smith, K. G., & Shalley, C. E. 2006. The interplay between exploration and exploitation. Academy of Management Journal, 49(4): 683-706.

Hayward, R. A., & Hofer, T. P. 2001. Estimating hospital deaths due to medical errors: Preventability is in the eye of the reviewer. Journal of the American Medical Association, 286(4): 415-420.

Heimann, L. 2005. Repeated failures in the management of high risk technologies. European Management Journal, 23: 105-117.

Jagsi, R., Kitch, B. T., Weinstein, D. F., Campbell, E. G., Hutter, M., & Weissman, J. 2005. Residents report on adverse events and their causes. Archives of Internal Medicine, 165: 2607-2613.

Johnson, P. E., Hassenbrock, F., Duran, A. S., & Moller, J. H. 1982. Multimethod study of clinical judgment. Organizational behavior and human performance, 30: 201-230.

Johnson, P. E., Moen, J. B., & Thompson, W. B. 1988. Garden path errors in diagnostic reasoning. In L. Bolc, & M. J. Coombs (Eds.), Expert System Applications. Berlin: Springer-Verlag.

Johnson, P. E., & Thompson, W. B. 1981. Strolling down the garden path: Detection and recovery from error in expert problem solving. Paper presented at the Seventh International Joint Conference on Artificial Intelligence, Vancouver, BC.

Kegan, R. 1994. In Over Our Heads: The Mental Demands of Modern Life. Cambridge, MA: Harvard University Press.

Klayman, J. 1988. Cue discovery in probabilistic envirnoments: Uncertainty and experimentation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(2): 317-330.

Klayman, J., & Ha, Y.-W. 1987. Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review, 94(2): 211-228.

35

Klein, G., Phillips, J. K., Rall, E., & Peluso, D. A. 2006. A Data/Frame Theory of Sensemaking. In R. Hoffman (Ed.), Expertise Out of Context. Mahway, NJ: Erlbaum.

Klein, G., Pliske, R., Crandall, B., & Woods, D. 2005. Problem detection. Cognition, Technology and Work, 7(1): 14-28.

Klein, G. A., Orasanu, J., Calderwood, R., & Zsambok, C. E. 1993. Decision Making in Action. Norwood, NJ USA: Ablex Publishing.

Langer, E. J. 1989. Minding matters: The consequences of mindlessness-mindfulness. In L. Berkowitz (Ed.), Advances in Experimental Psychology, Vol. 22: 137-173. San Diego: Academic Press.

Louis, M. R., & Sutton, R. I. 1991. Switching cognitive gears: From habits of mind to active thinking. Human Relations, 44(1): 55-76.

Mandler, G. 1982. Stress and thought processes. In S. Goldberger, & S. Breznitz (Eds.), Handbook of Stress: 88-164. New York: Free Press.

Mandler, G. 1984. Mind and Body. New York: W. W. Nortorn & Company.

March, J. G. 1991. Exploration and exploitation in organizational learning. Organization Science, 2(1): 71-87.

March, J. G. 1996. Continuity and change in theories of organizational action. Administrative Science Quarterly, 41: 278-287.

March, J. G. 2006. Rationality, foolishness, and adaptive intelligence. Strategic management Journal, 27: 201-214.

Nielsen, R. P., & Bartunek, J. M. 1996. Opening narrow, routinized schemata to ethical stakeholder consciousness and action. Business and Society, 35(4): 483-519.

Pearson, C. M., & Clair, J. A. 1998. Reframing crisis management. Academy of Management Journal, 23(1): 59-76.

Pronovost, P., Weast, B., Schwarz, M., Wyskiel, R., Prow, D., Milanovich, S., Berenholtz, S., Dorman, T., & Lipsett, P. 2003. Medication reconciliation: a practical tool to reduce the risk of medication errors. Journal of Critical Care, 18(4): 201-205.

Reason, J. 1990. Human Error. New York: Cambridge University Press.

Repenning, N. P., & Sterman, J. D. 2002. Capability traps and self-confirming attribution errors in the dynamics of process improvement. Administrative Science Quarterly, 47: 265 - 295.

Roberts, N. C., & Dotterway, K. A. 1995. The Vincennes incident: Another player on the stage? Defense Analysis, 11(1): 31-45.

Rudolph, J. W. 2003. Into the big muddy and out again: Error persistence and crisis management in the operating room. Boston College, Chestnut Hill, MA.

Rudolph, J. W., & Raemer, D. B. 2004. Diagnostic problem solving during simulated crises

in the OR. Anesthesia and Analgesia, 98(5S): S34.

Rudolph, J. W., & Repenning, N. P. 2002. Disaster dynamics: Understanding the role of quantity in organizational collapse. Administrative Science Quarterly, 47: 1-30.

Sastry, M. A. 1997. Problems and paradoxes in a model of punctuated organizational change. Administrative Science Quarterly, 42: 237-275.

Schiff, G., Kim, S., Abrams, R., Cosby, K., Lambert, B., Feldstein, A. S., Hasler, S., Krosnjar, N., Odwazny, R., Wisniewski, M. F., & McNutt, R. A. 2004. Diagnsotic Errors: Lessons from a

36

Multi-Institutional Collaborative Project for the Diagnostic Error Evaluation and Research Project Investigators. Rockville, Maryland: Agency for Healthcare Research and Quality.

Schwandt, D. R. 2005. When managers become philosophers: integrating learning with sense making. Academy of Management Learning & Education, 4(2): 176-192.

Sitkin, S. B., Sutcliffe, K., & Schroeder, R. G. 1994. Distinguishing control from learning in total quality management: A contingency perspective. Academy of Management Review, 19(3): 537-564.

Smith, S. M., & Blankenship, S. E. 1991. Incubation and the persistence of fixation in problem solving. American Journal of Psychology, 104(1): 61-87.

Snook, S. A. 2000. Friendly Fire: The Accidental Shootdown of US Black Hawks Over Northern Iraq. Princeton, NJ USA: Princeton University Press.

Starbuck, W. H., Greve, A., & Hedberg, B. L. T. 1978. Responding to crises. Journal of Business Administration, 9(2): 111-137.

Staw, B. M. 1976. Knee-deep in the Big Muddy: A study of escalating commitment to a chosen course of action. Organizational Behavior and Human Performance, 16: 27-44.

Staw, B. M., Sandelands, L. E., & Dutton, J. E. 1981. Threat-rigidity effects in organizational behavior: A multilevel analysis. Administrative Science Quarterly, 26: 501-524.

Sterman, J. 2000. Business Dynamics. Chicago: Irwin-McGraw Hill.

Sterman, J. D. 1994. Learning in and about complex systems. System Dynamics Review, 10(2-3): 291-330.

Sterman, J. D., & Wittenberg, J. 1999. Path dependence, competition, and succession in the dynamics of scientific revolution. Organization Science, 10(3): 322-341.

Strauss, A., & Corbin, J. 1994. Grounded theory methodology: An overview. In N. K. Denzin, & Y. S. Lincoln (Eds.), Handbook of qualitative research: 273-285. Thousand Oaks: Sage.

Suddaby, R. 2006. From the Editors: What grounded theory is not. Academy of Management Journal, 49(4): 633-642.

Torbert, W. R. 1991. The Power of Balance: Transforming Self, Society, and Scientific Inquiry. Newbury Park, CA: Sage.

Torbert, W. R., & Associates. 2004. Action Inquiry: The Secret of Timely and Transforming Leadership. San Francisco: Berret-Koehler Publishers.

Voytovich, A. E., Rippey, R. M., & Suffredini, A. 1985. Premature conclusions in diagnostic reasoning. Journal of Medical Education, 60(4): 302-307.

Wachter, R. M. 2004. The end of the beginning: Patient safety five years after "To Err is Human". Health Affairs, November(??): ??

Waller, M. J. 1999. The timing of adaptive group responses to nonroutine events. Academy of Management Journal, 42(2): 127-137.

Weick, K. E. 1993b. The collapse of sensemaking in organizations: The Mann Gulch disaster. Administrative Science Quarterly, 38: 628-652.

Weick, K. E., Sutcliffe, K., & Obstfeld, D. 2005. Organizing and the process of sensemaking. Organization Science, 16(4): 409-421.

37

Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. 1999. Organizing for high reliability: Processes of collective mindfulness. In R. I. Sutton, & B. M. Staw (Eds.), Research in Organizational Behavior, Vol. 21: 81-123. Stamford, CT: JAI Press.

Wu, A. W., Folkman, S., McPhee, S. J., & Lo, B. 1991. Do house officers learn from their mistakes? Journal of the American Medical Association, 265(16): 2089-2094.

Xiao, Y., & MacKenzie, C. F. 1995. Decision making in dynamic environments: Fixation errors and their causes. Paper presented at the Human Factors and Ergonomics Society 39th Annual Meeting, Santa Monica, CA.

Zsambok, C. E., & Klein, G. 1997. Naturalistic Decision Making. Mahway, NJ: Lawrence Erlbaum Associates.

38

APPENDIX

Figure A1: Stock and Flow Diagram of Model Structure

Documentation of Model Equations

(01) Abandoning Algorithm=Change Trigger*Treatments and Studies Completed/TIME STEP Units: steps/Minute (02) Accuracy of Diagnosis=IF THEN ELSE(Current Diagnosis=True Diagnosis, 1, 0) Units: Dmnl (03) Building Confidence=Initializing Confidence+Updating Confidence Units: Dmnl/Minute (04) Change Trigger=IF THEN ELSE(Share of Confidence<Confidence Threshold, 1, 0) Units: Dmnl (05) Changing Diagnosis=Change Trigger*Diagnosis Increment/TIME STEP Units: diag number/Minute (06) Conducting Treatments and Studies=Undiscovered Evidence*Fractional Rate of Studies Units: steps/Minute (07) Confidence in Current Diagnosis= INTEG (Building Confidence-Losing Confidence,Confidence in New Diagnosis) Units: Dmnl (08) Confidence in New Diagnosis=0.75 Units: Dmnl (09) Confidence Threshold=0.2

39

Units: Dmnl (10) Current Diagnosis= INTEG (Changing Diagnosis,0) Units: diag number (11) Degree of Exploitation=(Treatments and Studies Completed/Depth for Max Tx Effectiveness) Units: Dmnl (12) Depth for Max Tx Effectiveness=4 Units: steps (13) Diagnosis Increment=1 Units: diag number (14) Effect of Confidence on Doing Treatment=Confidence in Current Diagnosis^Strength of Confidence Effect Units: Dmnl (15) Effect of Confidence on Interpretation=Table for Effect of Confidence on Interpretation(Confidence in Current Diagnosis) Units: Dmnl (16) False Positive=1 Units: Dmnl (17) FINAL TIME = 45 Units: Minute The final time for the simulation. (18) Fractional Rate of Studies=Effect of Confidence on Doing Treatment*Propensity to Treat and Study Units: 1/Minute (19) Gaining Support for Alternative=Normal Fractional Exploration*Unexplored Potential+Iniitial Support for Alternative*Change Trigger/TIME STEP Units: Dmnl/Minute (20) Indicated Confidence=Interpretted Support*Degree of Exploitation Units: Dmnl (21) Iniitial Support for Alternative=0.05 Units: Dmnl (22) INITIAL TIME = 0 Units: Minute The initial time for the simulation. (23) Initializing Confidence=(Confidence in New Diagnosis/TIME STEP)*Change Trigger Units: Dmnl/Minute (24) Interpretted Support=Accuracy of Diagnosis*Weight on Accuracy+(1-Weight on Accuracy)*False Positive Units: Dmnl (25) Losing Confidence=Purging Confidence Units: Dmnl/Minute (26) Max Support for Untested Alternative=0.2 Units: Dmnl (27) Normal Fractional Exploration=0.5 Units: 1/Minute (28) Propensity to Treat and Study=1 Units: 1/Minute (29) Purging Confidence=Change Trigger*(Confidence in Current Diagnosis/TIME STEP) Units: Dmnl/Minute (30) Resetting Alternative=Change Trigger*Support for Alternative/TIME STEP Units: Dmnl/Minute (31) SAVEPER = TIME STEP Units: Minute [0,?] The frequency with which output is stored. (32) Share of Confidence=Confidence in Current Diagnosis/(Confidence in Current Diagnosis+Support for Alternative) Units: Dmnl (33) Strength of Confidence Effect=1 Units: Dmnl

40

(34) Support for Alternative= INTEG (Gaining Support for Alternative-Resetting Alternative,Iniitial Support for Alternative) Units: Dmnl (35) Table for Effect of Confidence on Interpretation ([(0,0)(1,1)],(0,1),(0.192661,0.95614),(0.321101,0.877193),(0.446483,0.723684),(0.5,0.5),(0.574924,0.254386),(0.685015,0.105263),(0.850153,0.0307018),(1,0)) Units: Dmnl (36) TIME STEP = 0.25 Units: Minute [0,?] The time step for the simulation. (37) Time to Update=3 Units: Minute (38) Treatments and Studies Completed= INTEG (+Conducting Treatments and Studies-Abandoning Algorithm,0) Units: steps (39) True Diagnosis=3 Units: diag number (40) Undiscovered Evidence=Depth for Max Tx Effectiveness-Treatments and Studies Completed Units: steps (41) Unexplored Potential=Max Support for Untested Alternative-Support for Alternative Units: Dmnl (42) Updating Confidence=(Indicated Confidence-Confidence in Current Diagnosis)/Time to Update Units: Dmnl/Minute (43) Weight on Accuracy=Effect of Confidence on Interpretation Units: Dmnl