United StatesDepartment ofAgriculture

Forest Service

Assessing Uncertainty in ExpertJudgments About Natural Resources

Southern ForestExperiment Station

New Orleans,Louisiana

David A. Cleaves

General Technical Reportso-1 10July 1994

SUMMARY

Judgments are necessary in natural resources management, but uncertainty about thesejudgments should be assessed. When all judgments are rejected in the absence of hard data,valuable professional experience and knowledge are not utilized fully. The objective of assessinguncertainty is to get the best representation of knowledge and its bounds. Uncertainty assessmentscan be used to compare alternative projects and risks, help prevent use of extreme decisionstrategies, help communicate and justify decisions, guide research, and establish monitoringprograms to improve learning.

Uncertainty assessment is the art and skill of judging the level of knowledge that backs upestimates. Use of structured processes, use of an objective analyst or facilitator, and developmentof rewards for honesty and introspection in professional judgment can increase the accuracy ofassessments. Techniques and performance aids are available for structuring decisions and uncertainelements, guarding against motivational and cognitive biases, and dealing with rare butconsequential events. Experience and increased understanding of decision analysis, artificialintelligence, and behavioral decision theory will also enable decisionmakers to make increasinglyaccurate assessments of uncertainty in professional judgment.

ACKNOWLEDGMENTS

The author thanks the following people who contributed to this publication:

Bill Bradshaw Dale McDanielSteve Derby Joan NormanJames Granskog Alan SalmonMinnie Haten James SavelandDenise Hutchinson Paul Slavic

contents

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1A CASE FOR ASSESSING UNCERTAINTY ................................. 1

Nature of Uncertainty and Human Judgment ............................... 1Uncertainty and Risk ............................................ 2

Uncertainty Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2ASSESSMENT PROCESS AND GUIDELINES ................................ 4

FormsofExpression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Verbal Expressions. ............................................. 4Numerical Expressions ........................................... 4

Assessment Process ................................................ 5Assessment Strategies ............................................ 5Role of the Analyst/Facilitator ...................................... 7Assessment Process Outlined ....................................... 7

Structuring the Decision and the Uncertain Element ......................... 8Decision Structures .............................................. 8Element Structures .............................................. 8

Response Frames for Assessments ......................................10Guidelines for Conducting Assessments .................................. 11Judgmental Biases ................................................. 12Modes of Judgment-Sources of Bias ................................... 13

Anchoring ........................................ . 13Reliance on Availability .............................. . 13Reliance on Representativeness ......................... . 14Reliance on Internal Coherence ......................... . 14Reliance on Hindsight ................................ . 14

Group Assessments ..................................... . 14Assessing Rare Events .................................. . 15Assessment Quality .................................... . 15

LITERATURE CITED.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

i

Assessing Uncertainty in Expert JudgmentsAbout Natural Resources

David A. Cleaves

INTRODUCTION

Those who make decisions about natural resourcepolicy usually rely on experts to interpret biological, social,and economic systems and to forecast the effects ofproposed policy alternatives. Similarly, natural resourcemanagers rely on professional specialists to predict thefuture and the consequences of decisions. Estimates bythese experts are sometimes regarded as accurate andcertain representations of the real world. However, asprofessional judgments and extrapolations from existingdata, such estimates are laced with uncertainty. The degreeand nature of this uncertainty is often not revealed or isdescribed only in vague, qualitative terms. Research andexperience in decision analysis have shown that uncertain-ty, although it may not be capable of being measured, canbe expressed in quantitative terms that can be understoodand used in decisions.

All natural resource management decisions involvesome uncertainty. Decisiomuakers are likely to developelaborate and costly alternatives or forgo important oppor-tunities in order to relieve or avoid vague uncertainties thatthey can feel but cannot articulate. Quantitative analysiscan indicate possible consequences of some--but not all--proposed actions. Moreover, quantitative analyses can beunacceptably expensive, or of poor quality, or not in formsusable in managerial and political decision processes.Even if available, quantitative analysis may not be anymore accurate or more useful than direct human judgment.The appearance of precision given by numerical outputs ofsuch analyses can mask the fact that the analyses and theirinterpretations reflect human judgment.

Given the limitations of analytical models and pub-lished studies, it may be necessary or desirable to directlyelicit judgments from scientists and specialists. Where thisis done, the scientist becomes an interpretive filter betweenthe body of scientific knowledge and the policy or decisi-onmaking process. Many sources of uncertainty can intlu-ence these judgments. Decisionmakers may be forced tochoose among alternatives that embody unexpressed judg-mental uncertainties. When uncertainties are not expressedproperly, those who develop or choose among alternativescan be misled. A working definition of uncertainty anda rationale for quantifying uncertainty are

presented, and general strategies and guidelines for estimat-ing uncertain quantities, values, relationships, and eventsare offered here.

A CASE FOR ASSESSING UNCERTAINTY

Nature of Uncertainty and Human Judgment

Judgment is a process of estimating, valuing, orchoosing--in essence it is a thinking process. Judgmentscan be expressed in various ways. They can be viewed asdata collected directly from human cognitive processes.The quality of judgments reflects the level of rationalityand scientific rigor in the judgment process. An expert mayhave technical understanding of processes and relationships,but there is no assurance that the expert’s judgment processwill follow the rules of rational thought or scientific reviewor that the judgment will constitute useful information.Rationality in judgment means that the person who esti-mates, values, or chooses thoroughly uses availableinformation and is aware of alternatives and their implica-tions and that the judgment is coherent, consistent withsimilar judgments, in agreement with general laws ofprobability, and understood by users. Scientific judgmentis rational judgment that has been challenged, has stood upto or been revised as a result of experience, and could berepeated if necessary.

“Soft” and “subjective” are terms often applied tojudgments that have not passed scientific muster. However,there are no absolutely “hard” or “objective” judgments.All judgments--even those we consider to be facts--have asubjective component. Theoretically, the more rational andscientific the judgment process, the better are the policyand operational decisions made.

Uncertainty is a condition of not knowing. All judg-ments are made with imperfect knowledge, and thus noestimate is completely accurate or perfectly predictive.Uncertainty can be about present conditions or about futureevents. The judgment process can hide or ignore uncer-tainty or it can seek to explicate it. When structuredprocedures for explicating uncertainty in judgment arefollowed, understanding of the judgment and its implica-tions is increased by opening the judgment up to produc-tive scrutiny and critique.

David A. Cleaves is principal economist in Evaluation of Legal, Tax, and Economic Influences on Forest ResourceManagement, U.S. Department of Agriculture, Forest Service, Southern Forest Experiment Station, New Orleans, LA 70113.

1

Finkel (1990) lists four sources of uncertainty:(1) Parameter uncertainty emanates from random and

systematic errors in measurement or estimation. Discus-sions of forest amenities, long-term projections of timberprices, estimates of effects of forest policies on employ-ment, and estimates of project costs and completion timesprovide many examples of parameter uncertainty.

(2) Model uncertainty results from imperfections inrepresentations of the real world--that is, in estimates ofrelationship among model parameters. For example, it isdifficult to model the causal links among various parts ofthe economy accurately; macroeconomists and regionalscientists struggle constantly to estimate changes in theserelationships and their influences on human welfare.Model uncertainty is present when biological responses offorest ecosystems to new silvicultural approaches andresponses of private landowners to changes in incentives orregulatory policies are estimated.

(3) Decision-rule uncertainty surrounds thequantification or comparison of social values and prefer-ences. These values can affect decisions profoundly but aredifficult to quantify and are subject to the vagaries ofpublic opinion. Participants in policy processes oftenunderstand what they don’t want more clearly than theyunderstand what they want, and their notions changerapidly. The setting of forest practice standards involvesselecting a level of risk that balances protection of theenvironment with maintenance of landowner rights andinitiative and feasibility of implementation. Comparisonsof such disparate elements can be so uncertain thatpolicymakers opt for simplistic regulations that may turnout to be ineffective, unduly restrictive, or difficult toadminister.

(4) Variability in the natural occurrence of modelparameters or decision variables across time, space, andhuman interactions is the classical notion of frequency andis amenable to statistical modeling and tasting. Forestplantation failures, prescribed fire escapes, historic stump-age prices, and many other parameters are often analyzedfor their variability. Even with complete certainty inparameter measurement, model representation, and deci-sion-rule specification, variability in parameters can makeoutcomes uncertain.

Uncertainty and Risk.-Risk is exposure to a chanceof loss. A full description of any risk includes cleararticulation and quantification of the exposure, chance, andloss dimensions. Such a description would be “a 5-percentchance of fusiform rust hitting a loblolly pine plantationwithin the next 10 years and causing a loss of 10 percentin growth and yield.” Here, the chance is 5 percent, theexposure is to our loblolly plantation over 10 years, andthe loss is 10 percent of growth and yield.

Evaluation of risk is becoming increasingly importantin natural resource and environmental protection decisions(Talcott 1992). Risk assessments are usually made up ofpoint estimates or averages. As mixtures of objective and

subjective estimates of quantities, frequencies, and othercomponents, these assessments are subject to their ownsecond or&r uncertainties. Quantitative risk assessmentshould include techniques for incorporating uncertainty intoregulatory decisionmaking.

Uncertainty Assessment

Assessment of uncertainty in professional judgment isnecessary in a world that demands quick answers tocomplex scientific questions. Gregory and others (1992)identified inadequate treatment of uncertainty in environ-mental assessments as one of five main problems reducingthe usefulness and applicability of environmental impactstatements required by the National Environmental PolicyAct. Informal and formal assessments of uncertaintyabound in the decisions about silvicultural strategies andforest protection, especially protection from fire. A recentanalysis of alternatives for management of late successionalforests in the Pacific Northwest (Johnson and others 1991)incorporated biologists, and managers, subjective estimatesof risks of extinction for groups of fish and wildlifespecies. The most notable and recent use of uncertaintyassessment is the report of the Forest Ecosystem Manage-ment Assessment Team (FEMAT), a group of scientistscommissioned by President Clinton to develop a plan forembattled Federal forests in the Pacific Northwest (USDAFS 1993). Uncertainty assessment procedures were used byFEMAT to elicit judgments about habitat viability forhundreds of old-growth and late successional plant andanimal species (Cleaves 1993).

In these and other examples, it is often the elicitationprocess more than the assessment itself that gives rise tocontention or threatens the credibility of the organizationor implementation of the decision. Problems in the processof eliciting judgments can produce inaccurate or incompre-hensible results. Armstrong (1978) provides a thorough andprovocative discussion of subjective judgment in themaking of long-range forecasts. Cleaves (1987), Cleavesand others (1987), and Saveland and others (1988) outlineimplications for eliciting subjective judgments in theconstruction of expert systems and forest protectionmodels.

Uncertainty assessment is the process of assessing anindividual’s or a group’s state of information about a givenoutcome or event. It involves telling not only what but alsohow much one knows. For example, one might assess theuncertainty in a fire expert’s judgment that a prescribed firewill escape under given conditions. Formal assessments ofuncertainty can be useful to those who make major deci-sions about policies and rules, capital investment, andstrategic planning. They can be useful when there isdisagreement about the problem deftition or the outcomesof proposed alternatives, where there is insufficient data forstatistical analysis, or both. A good assessment of uncer-tainty about an estimate accurately represents the

2

estimator’s degree of confidence in the estimate and fullytaps the estimator’s most rational and consistent judgmentin a form that can be used in the decision process. Assess-ments are most accurate when rigorous and systematicprocesses for eliciting judgments are employed. Theseprocesses are designed to control and compensate forinconsistencies and eccentricities inherent in humanjudgment processes. The effort expended in improvingthese judgments should be matched to the importance ofthe information. The uncertainty of estimates that arecritical in influencing choices that are economically orsocially important must be assessed with some rigor.

Decisiomnakers too often evaluate the quality of ajudgment solely on the basis of the estimator’s self-confi-dence or credibility as an expert. Some decisionmakersaccept expert opinion without questioning whether theyhave captured the true extent of the expert’s knowledge.Many of these decisionmakers have been disappointedwhen actual outcomes have differed from those predictedor when the public strongly disagrees with the expert. Atthe other extreme are decisionmakers who reject all expertjudgments as arbitrary and baseless guesses--and whosubstitute their own judgments for those of the better-informed experts.

Is it important to express the uncertainty in expertjudgments? A ‘yes” answer assumes that confrontinguncertainty leads to better decisions. Good decisions arebased on the full range of information available and areconsistent with the beliefs and preferences of thedecisiomnaker. Decisionmakers must deal with uncertaintyeven if it is not well described or measured. Manydecisionmakers try to resolve uncertainty by relying onhighly confident and specialized experts. Others ignoreuncertainties, treat uncertain events as if they were certain,hide uncertainties, gather information compulsively, orbuild in safeguards to ensure against negative outcomes.These strategies can misallocate intellectual, financial, andphysical resources if minor uncertainties get more attentionthan major uncertainties.

Formal uncertainty assessment can give decisiomnakersa positive attitude about confronting uncertainty. Thisbenetit is not obtained where the decision has already beenmade and where the uncertainty assessment becomes ajustification exercise. The benefit is not obtained whendecisiomnakers want to build in elaborate safeguards andcontingency plans against all types of uncertainty, andthese decisionmakers will likely ignore or circumventattempts to quantify and compare uncertain elements.Reward systems in organizations encourage confidence andcontrol; they do not benefit those who discuss what is notknown or controllable.

Another important benefit of assessing uncertainty isin relieving the “tension between analysis and action”(Finkel 1990). Having to identify, explain, and quantifydifferent sources of uncertainty can help decisionmakers

steer a rational path between “analysis paralysis” andimpulsive reaction. One strategy for responding to uncer-tainty is to defer action “until more information is avail-able” or “until the facts have been established.” Theopposite strategy is to take strong and immediate actionbecause “the future is too uncertain to permit inaction.” Intheir extremes, these approaches can lead to indecision orto frenzied overprotection, and they have too often becomeenduring strategies that polarize interest groups and blockcompromise.

Expressing uncertainty can also increase publicinvolvement and communication. In natural resource issues,many organizations participate in decisionmaking, andopportunities to miscommunication are many. If the publicis to participate effectively, the public must be able todistinguish facts from myths, values, and unknowns.Honest expressions of how little or how much is knowabout important points can help the public understand theissues. Moreover, the public may place greater trust inorganizations that express uncertainty candidly than inorganizations that claim perfect knowledge.

Quantification and comparison of uncertainty can helpguide information gathering and research. Collection andanalysis of information is costly and can represent amisallocation of resources. Without a guiding assessmentof knowledge gaps, data collection can evolve into asmokescreen to disguise decisions already made or sleightof hand that creates the illusion that uncertainties havebeen dealt with.

Uncertainty assessment as an ongoing process can alsoencourage organizations to monitor decision outcomes asa way of helping specialists and decisionmakers improvetheir judgments. Evaluation of the quality of judgmentrequires a refined ability to distinguish between choice andchance. Actual events provide feedback to specialists andsuggest ways in which specialists might adjust theirjudgments. However, without some rigor and documenta-tion in initial assessments of uncertainty, feedback fromactual events provides little basis for comparison. Asexperience accumulates, estimators can be rewarded forbeing candid about states of knowledge, anddecisionmakers can be rewarded for following gooddecisionmaking processes. According to Fischhoff (1990),organizations “must create (and demonstrate) an incentivestructure that rewards experts for saying what they reallybelieve, rather than for exuding confidence, avoidingresponsibility, generating alarm, or allaying fears.”

Finally, uncertainty assessments can be used as abasis for comparing proposed projects, risks, or policies.When described with point estimates or even averages,alternatives may appear equally uncertain or may notappear to be subject to any uncertainty. These conditionsoften lead decisiomnakers to make poor choices. Incorpo-rating uncertainty assessment into ranking of options canprovide a more complete perspective.

3

ASSESSMENT PROCESS AND GUIDELINES

Forms of ExpredonVerbuf Expressions.-Individuals u s u a l l y t r e a t

uncertainty as a feeling or belief but are usually not forcedto describe or quantify it. Common verbal expressionsindicating levels of uncertainty include:

l adapted from beyond reasonable doubt’ commonlyl conceivablel fair, even, or poor chancel frequentlyl highly likely (or unlikely)l high, medium, or low probabilityl incrediblel possiblyl probablyl rarelyl remotely possiblel seldoml usually.

Verbal expressions are most useful as indicators oflevels of uncertainty where differences between levels ofuncertainty are obvious. Words are easier to use thannumbers; they do not demand data or disciplined precision.According to Behn and Vaupel(1982), verbal expressions“reflect more than imprecise communication. They reflectimprecise thought. . .and mask. . unwillingness to thinkcarefully about uncertainty.”

As indicators of uncertainty, verbal expressions haveseveral major disadvantages (Beyth-Marom and others1985). First is their imprecision. “Likely” can meandifferent things to different people and even differentthings to the same person at different times (Capen 1984).For a particular event, “likely” may mean a probability ofoccurrence of 0.4 to 0.6 to one person and 0.9 to 0.95 toanother. “Beyond reasonable doubt” means from less than70-percent certain to some judges to IOO-percent certain toother judges. Jurors range from less than 50-percent toloo-percent certainty in their interpretation of “guilty bypreponderance of evidence.”

Second, the meanings of verbal expressions are veryspecific to the context. The numerical probability of a“likely” event in one situation may be quite different fromthat of a “likely” event in another situation. Third, verbalexpressions are not always used with sensitivity to smallbut potentially important changes in beliefs about events.An event may be called likely both before and after itsprobability doubles. Finally, words may confuse beliefabout the occurrence of an outcome with belief about thevalue of the outcome. For example, the statement that anecosystem has a “good” or “promising” chance of recover-ing from a natural disaster may be presented as a descrip-tion of relative likelihood, but it also reveals that theassessor considers recovery a good thing. The assessmentcould be more the result of wishful thinking than of a

dispassionate evaluation of uncertainty. Words are vagueenough to conceal human biases and evoke emotions thatcan persuade people to choose alternatives they would notchoose on a more rational basis.

Verbal expressions can be misleading unless therationale or the information base for the assessment isunderstood fully. Verbal expressions of uncertainty can beuseful but should be presented in a comparable scale.Repeated experience with verbal assessments can be usedto construct relationships between a probability scale andcommonly used words, as in figure 1 (Kent 1964).

Nume&af ~ressions.-Numerical expressions canbe understood by all users, allow comparison acrossdifferent elements of uncertainty, can convey small differ-ences in uncertainty, and separate a quantitative judgmentof an event’s probability from a qualitative judgment of anevent’s consequences. However, numerical expression ofthe uncertainty requires more discipline, analysis, training,and expense. Because the vast majority of uncertainties

VERBALEXPRESSION

AlmostCertainly

Highly Likely

Very GoodChance

Probable

Likely

Probably

We Believe

Better ThanEven

We Doubt

Improbable

Unlikely

Probably Not

Little Chance

AlmostNo Chance

Highly Unlikely

ChancesAre Slight

0 IQ 20 39 40 50 90 70 80 90 100

ASSIGNED LIKELIHOOD (percent)

Figure I.--Proposed relation between verbal expressionand assigned likelihood adapted&m (Kent 1964).

4

are not critical, numerical assessments should not bemade unless precision is needed.

Figure 2 portrays several options for graphicallydisplaying numerical assessments of uncertainty. Not all ofthese forms are equally comprehensible, so users shouldtest them thoroughly and adopt them with caution.

Forms of numerical expression differ greatly in theirinformation content and difficulty of application. It is oftensimplest and most useful to assign events or values to“high, “medium,” and “low” levels of uncertainty.Ranges can be specified as intervals that capture the “true”value of the estimate with some level of confidence, say 90percent. A wide interval indicates a high level of uncer-tainty. For example in figure 2a, the range-and-box showsthe extremes, and the central portion (the box) contains afixed percentage of the possible values.

Other numerical expressions are used to describe thelikelihoods of particular values or events. These are calledsubjective, personal, or judgmental probabilities or likelihoods.These probabilities can be treated like statistical probabilitiesin calculations in order to combine or isolate elements orto show how one event is conditional on another. Ajudgmental probability is not a frequency per se, but ratherrepresents the assessor’s degree of belief in a possibleoutcome. Assessors can assign judgmental probabilities toevents that have true probabilities. Judgmental probabilitiestreat uncertainty as a property of knowledge about eventsor outcomes rather than as a property of events themselves.

Judgmental probability uses the assumptions and rulesof probability theory to model uncertainty. Under thisframework, individual probabilities are greater than 0 andless than 1, and all probabilities in a set (the probabilitiesof all possible events) sum to 1. The probabilities ofindependent events can be multiplied together to give anestimate of the probability that all events will occur together.These jointprobabilities are always less thantheir individualcomponents. Events that are interrelated can be modeled asconditional probabilities.

Where there are multiple events or values, uncertaintyassessments can be expressed as distributions. The simplestdistributions are histograms (fig. 2b) and pie charts (fig. 2~).Distributions can also be expressed as probability densityfunctions (the familiar bell-shaped curve) (fig. 2d), or ascumulative density functions (fig. 2e) that show ranges oflikelihoods that specific values will not be exceeded.Sometimes it is useful to estimate fixed positions in anuncertain distribution. Quartile points are the most commonlyused faed positions, which require an estimate of the medianvalue, the extreme highs and lows, and the midpoints betweenthe extremes and the median. A simplified form is the singlepercentile value, the 10th or the 90th percentile, that focuseson particular values that are critical in the decision process.

Another form of numerical expression of uncertaintyis the index that describes in a single measurement the overallvariation in the distribution. Such indices include the variance,the standard deviation, and the coefficient of variation(standard deviation as a percentage of the mean).

Numerical expressions of uncertainty can presentdifficulties, however. The more sophisticated the measurementof uncertainty, the more difficult it is for assessors to estimateit and for decisionmakers to comprehend it and incorporateit into their decisions. Percentile values provide only partialinformation about the range of possible outcomes and canbe misleading if they are not fully understood by bothassessors and decisiomnakers. Some factors in decisions areso vague that they are difticult to think about, much less breakdown into discrete events or values. However, in attemptingto define and determine the uncertainty of such factors, bothassessor and decisiomnaker may learn more about the elementand clarify what they do and do not know.

Assessment Process

Judgments of uncertainty are predisposed to many humanbiases and inconsistencies. However, a structured processfor eliciting these assessments can help guardagainst biases,create more consistency, and make assessments easier tocommunicate, understand, and repeat. Decisiomnakers canhave confidence in assessments that are demonstrablyunbiased, consistent, and comprehensible.

There are no right or wrong uncertainty assessments;the assessment process is designed to elicit information aboutthe state of knowledge, not to elicit a prediction or acommitment. Over time, comparing assessments with actualoccurrences can improve a person’s ability to modeluncertainty. However, the major purposes of uncertaintyassessments are to codify judgments and to helpdecisionmakers make choices that are consistent with theirown feelings and knowledge.

Assessment S&ate&~--Fir&e1 (1990) describes twoapproaches to the assessment task. The “bottomup” approachdecomposes the uncertain element into subelements. Theuncertainty of each subelement is assessednumerically. Thecomponent assessments are then combined--usuallymathematically--into an overall assessment of uncertaintyfor the element. This approach offers natural checkpointsfor testing overcontldence and other forms of bias. Wrestlingwith the subelements and the way they fit together can showwhere information and research activities are needed andencourage a more complete undemtandmg of the system beingmodeled. Uncertainties of subelements can be assessed byappropriate specialists, strengtheGg the technical foundationand credibility of the overall assessment. The bottom-upapproach also has disadvantages. It requires heavycommitments of time and can magnify systematic biases

5

(a)7

I ! ! ! ! ! ! I I ! ! ! ! ! I ! 11 ! I0 2 4 6 8 10 12 14 16 18 19

INCHES OF SNOW

0.25

_ 0.20

1; 0.15

ii) 0.10

:0.05

0.00O-2 2.4 4.6 6.6 6.10 10.12 12.14 14.16 16-16

INCHES OF SNOW

7% 10%INCHES

OF SNOW

14% m o-2a 2-40 4.6m 6-8a 8.10[171 10-12a 12-14123 14.16

36%

0.18-r

( )e

0 2 4 8 8 10 12 14 16 18 21INCHES OF SNOW

I:::::::::::::::::::::::::::::10 2 4 6 8 10 12 14 16 18 20 22 24 28 28 31

INCHES OF SNOW

0 2 4 6 8 10 12 14 18 1INCHES OF SNOW

Figure 2.-Options for displaying numerical expressions of uncertainty adapted from (Ibrekk and Morgan 1987).(a) box and range plot, (b) histogram, (c)pie chart, (d) probability densityfunction, (e) cumulative densityfunction.

6

in the subelement assessments. Assumptions about relation-ships among the components are very critical; errors inmathematical equations can negate rigorous efforts inassessment of the individual parts. Finally, intenseconcentration on smaller elements can lead to unprofitabledebates about unimportant questions.

The “top-down” approach seeks an overall assessmentfor the main element without breaking it down. This approachis easier, less time consuming, and avoids many unresolvedscientific controversies. However, it is a daunting task todeal “all at once’ with a complex phenomenon for whichthere is little direct experience or data. Experts are often sospecialized that they have no feel for broader questions.Furthermore, some elements are just too complex toassess directly.

The bottom-up and top-down approaches can be hybridizedby cross-checking bottom-up assessments provided byscientists and technical experts with top-down assessmentsby managers, planners, or others. The first group contributesscientific and technical perspectives, while the second groupcontributes their unique observational experience andmanagerial perceptions. The sensitivity of the overallassessment can be checked against the rigor and precisionbeing required in the bottom-up tasks.

Role of the Ana&&FW&tor.-Uncertainty assessmentsshould be led by one or more skilled, analyst/facilitators(A/F’s). Assessment is very difficult without this assistancebecause assessors can become overwhelmed with the taskand can be unaware of biases tbat creep into their judgments.The A/F leads the assessors (scientists, specialists, and/ordecisionmakers) through the process, motivating them torecognize and deliver useful and bias-free judgments. TheA/F knows the larger decision context and how the assessmentresults will be used, and can match assessment techniquesto the personalities of the assessors and the characteristicsof the uncertain element.

The A/F should challenge the assessors to uncover newsources of information and to confront inconsistencies. TheA/F can provide continuity across assessments by differentpeople and disciplines and can mediate disagreements duringgroup assessments. The AIF can also encourage assessorsto persevere through ambiguities and tedium, and can providefeedback for improving assessment skills.

The A/F can organize the process to make efficient useof the assessors’ time and efforts, and can document theprocess for justification to decisionmakers and the public.An A/F can also help translate assessors’ responses to

decisionmakers and the public and vouch for the assessment’sprocedural rigor.

Assessment Process OurlineR--Figure 3 sketches anassessment process. This process was designed to producenumerical assessments of uncertainty (McNamee and Celona1989, Merkhofer 1987, Spetzler and Stael von Holstein 1975),but it could be a framework for producing assessments inany form.

During the motivation phase, the AR establishes rapport withthe assessor and describes how the assessment fits into theoverall structure of the decision(s). The A/F also orientsthe assessor to the basic goals and philosophy of uncertaintyassessment, emphasizing that there are no right or wronganswers and assuring confidentiality of responses. This mayalso be a time for orientation to fundamentals ofprobability.

During the structuring phase, the A/F and the assessordefine clearly the uncertain elements to be assessed, choosean overall strategy for approaching the task, and organizethe work. They also select a form of expression and anappropriate scale. The AR helps the assessor explore theimplications of the decisions that are to be made and uncoverhidden assumptions and feelings that could bias the assessmentof uncertainty.

During conditioning the A/F and assessor list sourcesof data, research reports, or other information that could beused as background for the assessment. They also list keyassumptions that will serve as guideposts if the A/F and theassessor become distracted or gridlocked. The All? describespossible biases that may affect the assessment and explainstechniques that will be used to guard against them. The A/Fasks the assessors to recall significant events or organizational

Motivating

\Structuring

\Conditioning

\Encoding

\

Verifying

\

Communicating

Figure 3.--Process for assessing uncertainty.

7

standad or rules that might unduly intluenoe the assessmentresults. The A/F also encourages the assessor to talk aboutor write out descriptions of extreme values and scenariosabout the uncertain elements to widen the assessor’s thinkingbefore actual encoding begins.

Conditioning is also the time for practice in assessingthe uncertainty of common elements (unrelated to thedecision). The assessors estimate subjective probabilities andranges of probabilities, which are compared to actual data.These tests provide feedback and instill confidence inassessment skills.

During encoding, the A/F leads the assessors througheach of the uncertain elements, using any of a number oftechniques to elicit their responses. The A/F reviews theresults as they are produced and looks for inconsistencies,biases, and disagreements between assessors, often repeatingthe assessments using a different technique to check for furtherinconsistencies.

During verification the A/F shows the assessment resultsto the assessors and gets agreement that the results representthe assessors’best judgments. Assessors are asked to provideadditional justification for extreme probabilities or likelihooddistributions with irregular shapes. The assessments cansometimes be compared with existing data for similar typesof elements or with assessments of the same element by otherassessors. Sometimes assessors will adjust their judgmentsto reflect this “outside” information. In most cases, however,outside data is not available.

During the communication step in the process, the resultsare delivered and explained to the decisionmakers. The A/Fand the assessor help the decisionmaker understand theassessment’s implications. The A/F may have to furtheranalyze the results, or restructure the decision based on therange of outcomes implied by the assessment. Accordingthe Finkel(l990), the decisionmaker should understand fourthings: (1) implications of replacing single estimates withuncertainty assessments, (2) the costs of over-estimating orunderestimatinguncertainties,(3)sensitivityofthedecisionto unresolved controversies among specialists or scientificexplanations, and (4) implications for further data gatheringor research about the elements.

Structuring the Decirion and the Uncertain Element

Understanding the decision is a prerequisite to providinguseful assessments of uncertainty. Also, the uncertain elementsmust be defined fully and organized so that people can judgethem. Even complex and apparently chaotic problems canbe taken apart and arranged so that there is a commonundemtamhng of the assessment task among the participants.Several techniques, or decisionmaking aids, can assist thisstructuring process.

Decision Structures.--Decision trees (McNamee andCelona 1989, Merkhofer 1987, Morgan and Hem-ion 1989)are tools for displaying alternatives and the results of each

8

altemative’s interaction with important sources of unceiGnty(fig. 4). A decision tree is composed of decision nodes, whichrepresent sets of decision alternatives, and event nodes, whichrepresent sets of outcomes of the sources of uncertainty.The probability of events are described at the event nodes.The end points of the decision tree are the outcomes of variousinterplays of actions and events. These consequences are eithercompletely or partially out of the decisiomuaker’s control.

Outcome BX/

EVENT X(.7)

,,.!z@( >LlkelihoodsEVENT Y(.3)

\ Outcome BY

Figure 4.-Decision tree.

Decision trees can be analyzed for optimal decisionstrategies, the implications of mistakes in judgment, and formonetary values of information that might support revisionof the uncertainty assessments (probabilities). They can alsohelp identify elements that are most critical to decisions.

Influence diagrams (Shachter 1987) present flowchartlikedepictions of interactions of decisions and the uncertainoutcomes of elements without imposing decision treehierarchies or sequences (fig. 5). Influence diagramming canshow how probabilities can depend on other probabilities,other decisions, and revealed outcomes. Influence diagramscan then be turned into decision trees and analyzed.

Eihent Structures.--Assessment trees are devices forvisualizing complex events as hierarchical branching patternsof component events or values (von Winterfeldt and Edwards1986). They can be helpful in bottom-up uncertaintyassessments. Unlike a decision tree, an assessment treecontains no decision points, but it suggests possibleinterventions that might change the overall outcome. Theprobabilities assigned to branches can be combinedmathematically to give an overall rating or probability. Onespecial approach to assessment trees, the analytic hierarchyprocess (Saaty 1988) uses paired comparisons to integrate

-

vDECISION

3

A

UNCERTAIN

Figure 5.--Znfluence diagram.

Likelihood (P)

of element A

Likelihood of Likelihood of

element 6 element C

outcomes outcomes

Joint likelihood

of sequence

W,)

‘(?initiate

event

TIT P ( A ) * P(B,) * P(C2)

p(B2) P ( A ) * P ( B ) * P(c )P(C2)

2 2

Figure 6.--Event tree.

9

tangible and intangible elements, to analyze their relativeimportance mathematically, and to check for logicalconsistency.

The premise of assessment trees is that uncertainties ofcomponent events or values are easier to comprehend andassess than uncertainties of larger elements. Sensitivity testingof assessment trees can indicate which component assessmentsare more critical and should get more attention in the encodingprocess. There are four major types of assessment trees(figs. 6, 7, 8).

Event trees (fig. 6) start with an initial event and showwhat events or states of the system it might lead to. Eachsubsequent event leads to its own set ofpossible final eventsor conditions. The uncertainties of the various branches arestated as probabilities, and the probabilities of the branchesleading to the individual end conditions are multiplied togive probabilities for the end conditions. Event trees can beused to develop logical scenarios of easily imagined eventsand can reveal inconsistencies in thinking during theassessment process.

Fault trees (fig. 7) start with events or conditions andtrace their possible causes or precipitating events. “And” nodesindicate that all lower level events must occur before thehigher level event can occur. “Or” nodes indicate that any

P(E) = P(C) + P(D)

p(c) = ‘“*pikg$-iJANDH

Level A Level Battribute attribute

Level Cattribute

Site preparation

Investment IStand maintenance

WBo*

Insect . . .l Risk

Disease . . *Windthrow

Stand structurel Ecosystem Woody material

sustainability(Soil structure

Figure &-Value tree using an exumple of silvictdtural strategyevaluation.

LIKELIHOOD EXPLANATORYLEVEL FACTORS DATA

l Very lowr i s k /. d=s

l Low risk

l Medium +risk

l Relativelyhigh risk

l High risk

l Extremerisk

l Difficulty ofburn

l Weather

l Fuels

l Fuels inadjacent stands

l Suppressionequlpment

l Age distribu-tion of crewmembers

a Crew members’years together

l Crew members’famiiiarlty withsituation

l Time since lastfire

Figure 7.-Fault tree using an example of species extinction.Figure 9.--Inference bee using an example of likelihood&

prescribed fire will escape.

10

of the lower level events is sufficient to cause the higherlevel event. Probabilities of higher level events can becalculated by multiplying the probabilities at the “and” nodesand adding the probabilities at the “or” nodes.

Value trees (fig. 8) organize attributes, goals, or valuesinto in&mediate attributes and finally into tangible measure-ments. Each of the branches in a set is weighted accordingto its importance in fulfilling the higher level attribute. Theattribute “risk,” for example, could be broken into subattributessuch as dread, voluntariness, surprise, and manageability.Value trees can help identify pivotal factors in the decisionsand help assessors understand their own preferencesand biases.

Inference trees (fig. 9) start with hypotheses or scenariosthat are as yet unobserved. Preceding these hypotheses arelayers of events that explain the hypotheses or the scenario.At the lowest levels of the tree are sources of observabledata. Probabilities of the hypotheses are assessed after dataand observations are fitted to the structure of the tree.Inference trees can help resolve disagreements betweenassessors and encourage use of the existing knowledgeand data.

Response Frames for Assessments

Techniques for eliciting assessments of uncertainty differin their response frame--the form of numerical expressionand the approach to the estimate. Some techniques ask forprobabilities for a given outcome; some ask for outcomesat given probability levels; others ask for both. Sometechniques ask that a probability be estimated directly, whileothers compare the likelihood of the outcome to thelikelihoods of others. Some techniques are better for assessinguncertainties of discrete events and others for estimatingdistributions of continuous variables. Various techniques arecovered well in Finkel(l990), Kahneman and others (1982),Merkhofer (1987), Morgan and Henrion (1989), Spetzler andStael Von Holstein (1975), and Von Winterfeklt and Edwards(1986). It is common to assign risk levels (sometimes inprobability format) to future scenarios. Here, the assignmentof probabilities to discrete outcomes or scenarios is discussed.Uncertainty assessments for continuous distribution are moreelaborate and are covered in each of the references cited.

The indirect response frame asks for outcomes for afixed probability and is generally preferred over directprobability assessment. Many assessors are not comfortablein directly expressing probabilities, odds, or chances.

The usual procedure for indirect responses is to askfor 10th and 90th percentile and median outcomes. The analystoften uses props to help the assessor visualize the probabilityrelationships. The assessor compares probabilities withproportional slices of a pie or with segments of a horizontalbar. The entire pie circle or length of a bar represents aprobability of 1. The slice or segment is adjusted until theassessor feels it represents the correct likelihood. Both indirect

and direct assessments are very msceptible to overwntidencebias, so the analyst should word questions carefully andencourage assessors to imagine extreme circumstances.

The direct response frame can be effective after theassessor gains experience. Probabilities of less than 0.10 areusually handled better with direct estimates of probabilitiesor odds. A response can be expressed as an absolute number(0.20), a percentage (20 percent), a fraction (1 in 5), or evenas the logarithm of a probability or odds ratio. Probabilitiesof rare outcomes are more easily expressed in fractions suchas “1 in 100” or “1 in 1,000.”

Analysts/facilitators can check assessments by shiftingfrom one response frame to the other. If the initial assessmentwas in terms of direct probabilities, the A/F can ask fordescriptions of events at key probability levels like the 50th(median) or the 10th and 90th percentiles. Another checkis to change the visual prop or use a different analyst. Theseprocedures help identify inconsistencies and build con&lencein the assessment results.

Guidelines for Conducting Assessments

1. Use an analyst/facilitator. Select an A/F who is patient,impartial, and experienced in helping people throughjudgment tasks. The A/F should not be a decisionmaker,staff specialist, or one of the assessors. Beware of A/Fis who try to fit answers to particular analytical forms,even when the assessors do not seem to think in thoseterms. Use more than one A/F’s to mitigate fatigue andprovide constant vigilance against judgmentalinconsistencies. The A/F should u&r&and the decisionthoroughly but should not advocate any alternatives.

2. Select assessors carefully. Expertise in the subject mayensurethattheuncertainelementis lldX&OdbutdoeS

not guarantee high-quality assessments. Look for peoplewho have had experience making judgments andreceiving feedback in managerial and political decisions.Check their understanding of basic rules of probabilityand their willingness to express their judgments innumerical forms. Beware of “experts” who may haveulterior agendas or preconceived preferences forparticular outcomes. Assemble a portfolio of assessorswho can cover the technical territory and wumerbalanceeach other’s biases and weaknesses in assessment skill.

3. Create an environment where the assessor(s) can focuson the task. Remove distractions and any referencesto the decision that could bias the assessors’ thinking.

4. Relieve assessors’ fears. Emphasize that judgments ofprobability are no more difficult or dangerous than anyof the judgments of condition or quantity that theassessors make every day. Assure them that there areno wrong answers. Judgments of complete uncertaintyare perfectly acceptable and are valuable in the decisionprocess. Assessors should try to ensure that they havegiven their best judgments. They should be rewarded

11

5.

6.

7.

for their honesty and effort and not for displaying highconfidence in particular outcomes. Promise and deliveranonymity and confidentiality.Make efficient use of assessors’ time and efforts. Onlythe uncertainty of the elements most important to thedecision should be assessed. Isolate these criticalelements before beginning the assessment process.Define the uncertain element as specifically as possible.The definition should include all outcomes that couldactually occur and should exclude all others. Involveassessors in the definition process. Brainstorm“possibility lists” before the assessment begins to avoiddiscovering “new” outcomes while the numericalassignments are under way. Check element definitionsfor consistency with the decision structure. Ask assessorsto describe the element in their own words. All membersin a group assessment should agree on the definitionand be able to provide examples to one another. Chal-lenge assessors to think about the complements of singleoutcomes and to probe for possible outcomes that mightbe omitted. The clairvoyant test (Spetzler and Stael VonHolstein 1975) is a good check for completeness andclarity in outcome definition. In terms of this test, theclairvoyant knows all but only in the form of facts andcannot make interpretations or inferences. If aclairvoyant could identify the actual outcome witha true or false reply or without requesting clarification,it is well defined.Be prepared for problem elements. Growth rates andscenarios present special pitfalls and difficulties. Forexample, people find it diEcult to assess the uncertaintyof estimates of compounded quantities. It is better toask for the uncertainty surrounding an actual value atthe end of a specified growth period and calculate theimplied rate of growth (McNamee and Celona 1989).The probabilities of conjunctions of events are usuallyoverestimated because descriptions of such conjunctionsmake them seem more plausible. Likewise, probabilitiesof single events or scenarios with less detail are oftenunderestimated. Test scenario-type outcomes with simpleinference trees or questioning that encourages theassessors to imagine alternatives. It is easy to assumethat assessors understand what information is containedin a probability distribution or in parameters such asthe mean or the variance. Assessors may not know howthe mean, median, and mode differ or that distributionscan have different shapes. People tend to shy away fromasymmetric distributions because of long conditioningwith the normal bell-shaped curve, but many uncertainelements have their most consequential events in thetails (low probabilities). Leave assessments of continuousvariables to experts who know both the subject matterand the mechanics of probability density functions.Always cross-checkassessments of distributionparam-eters against estimates for specific outcomes. Peopletend to underestimate the uncertainty of outcomes thatare observed frequently and often assess the likelihoodof unique events poorly.

8.

9.

10.

11.

12.

13.

Describe outcomes in terms that are meaningful to theassessor and to the decisionmaker.Separate the task of describing the outcomes or eventsfrom that of encoding the probabilities. Asking assessorsto do both things at the same time makes their workharder and can cause assessors to overlook possibleoutcomes.Use a mixture of assessment formats, scales, andquestioning protocols with the same assessors ratherthan relying on only one. Cross-check results andaverage them if differences cannot be resolved.Challenge assessors to explain their rationale and toconsider extremely likely and unlikely outcomes. Askfor estimates for extremely unlikely outcomes first andthen obtain estimates for more likely outcomes. Peopletend to “anchor” on initial estimates unless they areassisted in recognizing the true variability in outcomes.DonY discourage low levels of confidence or wide rangesin credible intervals. A high level of uncertainty is alegitimate response, reflecting a true lack of predictabilitythat should be considered in the decision process. Highlevels of confdence (narrow ranges) are not necessarilygood, because confidence can emanate just as easilyfrom illusions as from accurate representations of theknowledge base or future events.Check estimates against any records of similar elements.Don’t overlook similar types of events or outcomes andany recorded experiences. Ask assessors to reconciledifferences between their assessments and historicalrecords.

Judgmental Biases

Assessment bias is a discrepancy between the assessor’sestimate and an accurate numerical description of the assessor’sknowledge. Calibration is the degree of agreement betweenthe assessment and the actual statistical probability of anoutcome. Both assessment bias and genuine lack of knowledgecontribute to poor calibration.

In many instances, calibration cannot be determinedbecause the relevant probabilities are unknown. However,the A/P should strive to obtain the best and most justifiableassessments possible. Assessment processes should be adjustedto correct for modes of judgment that give rise to biases.Bias can result from errors in defining the assessment taskor from the thinking processes of the people involved.Although the following discussion concentrates on assessorbiases, much of it could apply to equally serious biases ofA/F’s or decisionmakers.

Task bias occurs when the assessment task is not welldefined, and when assessors do not understand what theyare being asked to evaluate, but are reluctant to admit it.They provide answers anyway, often out of a sense of dutyor pride. Task bias can be prevented when A/P’s and assessorsknow exactly what is going to be done and exactly whatinformation is needed. The assessors should understand the

task so well that they can explain it clearly to others. Definingthe assessment task may take 60 to 80 percent of theassessment time, but it is worth the effort.

Conceptual bias can be motivational or cognitive.Motivational bias results from the assessor’s values orperceptions of rewards and punishments associated with theassessment results. Assessors may be predisposed in favorof one decision alternative and may unconsciously orienttheir assessments to support it. Personal feelings about peopleor organizations may also cloud assessors’ judgments. Somespecialists may be accustomed to being rewarded for beingconfident about proposals and inspiring this confidence inothers; they may find the process of expressinguncertainty difficult.

Disciplines often imply pledges to anticipate certainevents; to the specialist, these events may loom larger becausethey are a central focus. Experts often overestimate thelikelihood of dire events, especially when they perceive thattheir predictions are bases for action. They would rather bewrong by predicting events that do not happen than by failingto predict events that occur. For example, ecologists whofocus their work on ecosystem fragility may overestimatethe likelihood of an ecosystem disaster. Engineers, focusedon controlling natural systems, might underestimate theprobability of the same event.

Managers, specialists, and planners usually giveassessments that are too narrow, but for different reasons.Managers tend to concentrate on results as “done” (even beforethey are actually done). Specialists try to fulfill expectationsthat they are knowledgeable. Planners try not to disagreewith in-house predictions or assessments that have alreadybeen made. Some assessors may rely on informationthat looks technical because they think it is objective, butit is actually poor in quality or irrelevant to their tasks.

There is no cure for motivational bias, but it can becombatted in several ways. One approach is to divide theuncertain element into subelements, creating enough detailso that assessors do not know how their answers will servetheir self-interests. Assessors can also be reminded directlyabout motivational biases and asked to recognize and listsymptoms in their own thinking. The A/F should listen forsigns of wishful thinking or inappropriate pessimism oroptimism and should challenge assessors to explain theirassessments. Be alert for words such as “should” and “ought”that may indicate strong preferences for certain outcomes,and ask about personal experiences that could influence theassessor. Assure assessors that their work is confidential orthat they are otherwise protected from reprisal or reward.Assembling a mixture of assessors who may hold offsettingmotivational biases may help.

Cognitive bias is introduced by the way in which theassessor psychologically processes information. The mostcommon cognitive bias is to be overconfident, that is toestimate intervals or probability distributions that are toonarrow. This is also called conservatism. Other biases includepredicting events that happen at the same time (conjunctive),

insensitivity to available data about frequencies of similarevents (base rates), expecting sequential patterns of high andlow results to respect themselves, overestimating humancontrol, and mistakenly perceiving of patterns in randomphenomena. Bazerman (1986), Dawes (1988), Kahnemanand others (1982), and von Winterfeldt and Edwards (1986)provide good explanations of these and other biases andcorrective procedures.

Modes of Judgment-Sourcer of Bii

People judge uncertainty as they judge distance--byrelying on cues and landmarks that simplify complexcomputational tasks. These intuitive judgments become secondnature in various modes of judgment or subconscious heuristicprocesses(Kahnemanandothers 1982,TverskyandKahneman1974). Judgment is necessary for survival, but excessive orinappropriate reliance on specific modes of judgment canlead to serious cognitive biases. The A/F and the assessormust recognize such causes of bias. Important modes ofjudgment include anchoring, reliance an availability, relianceon representativeness, reliance on internal coherence, andreliance on hindsight.

Anchoring.--When estimating an uncertain outcome,the individual searches for an approximate starting point.If asked to provide a range, the individual adjusts from theinitial estimate. Two problems can beset this process. Firstis the possibility of a misleading start& point. With especiallydifficult or vague uncertain elements, assessors are readilytempted to adopt any suggestion--an organizational forecast,a strategic plan goal, a performance standard, or other value.Second is the common tendency to make only smalladjustments from the starting point, which acts as an anchoron the assessment. If information that is contrary to the initialestimate is provided, the individual usually ignores it. Thisproblem is worse with an easy assessment task, becauseassessors are very familiar with the element. These problemscan lead to inappropriately precise assessments of uncertainty.

The A/F should encourage assessors to fully consideroutcomes and probabilities beyond the ranges given in initialestimates. An important approach is to ask for extremeestimates first, regardless of response mode or technique.Asking first for the “most likely” or “best” or “average”estimate promotes anchoring and insufficient adjustmen& andresults in overconfidence that is hard to combat later. Whenfocusing on very rare or very frequent events, it may benecessary to ask first for outcomes at intermediateprobabilities, say 0.25 and 0.75.

Reliance on Availability.--Events that are more easilyremembered are usually judged to be more probable. Peopleforsake quality of information for convenience. Familiarity,saliency, recency, and imaginability influence judgments morethan statistical patterns do because they influence what isavailable from memory. Information that is ordered, mdundant,imaginable, sanctioned, or consistent with a person’s

13

worldview is more likely to be stored in memory and recalled(Kahneman and others 1982). The A/F should probe forexperiences that may have receded in memory and shouldpresent extreme outcomes as straw men, asking the assessorto imagine that they have already occurred and to explaintheir occurrence. The A/F shouid pay attention to howoutcomes are sequenced in the assessment questioning. Peopleoften grossly overestimate or underestimate values followingextreme outcomes because they ignore the normal regressiontoward the mean. For example, they may expect an extremelyhot, dry year to be followed either by another dry year orby an extremely wet year when an “average” year is actuallymore likely.

Assessors should also be warned about conjunctivefallacy--the tendency to overestimate the likelihood ofscenarios composed of several events occurring together.For example, the conjunction of a bad tire season, an insectoutbreak, personnel shortages, and a budget shortfall maybe a vivid and plausible scenario, but the likelihood of allthese events occurring together is quite small. Scientists whohave been trained to look for connections among elementsin ecological or economic systems can sometimes falselyimagine patterns in what are largely random coincidences.“Worst-case” scenarios, formerly required by the Councilon Environmental Quality (CEQ) in environmental impactanalyses were notoriously prone to conjunction bias.Decisiomnakers were thought to pay more attention to thevividness of the scenarios than to the extreme smallness ofthe probabilities that the scenarios would occur. The CEQabandoned the requirements for worst-case analysis in 1986.

Reliance on Representativeness.-People tend to judgeevents by their degree of similarity to other familiar eventsor to some stereotypic image. They extrapolate occurrencesfrom even a small sample of events to the event they arebeing asked to assess, becoming insensitive to sample size,the reliability of the evidence, or their own knowledge ofwhat could cause the event. Representativeness results instereotyping outcomes, as in imagining disastrous conse-quences where a few characteristics of a widely familiardisaster are present. Objective information may be forsakenfor vivid detailed descriptions of what are actually rare events.Media coverage of disasters, tragedies, and scandals or otherextremes (good or bad) can easily influence assessments.

The A/F should encourage the assessor to considerpublished data for similar uncertain elements and outcomes,and ask assessors for descriptions of events that seem to drivetheir judgments in the opposite direction. Too much talk aboutthe “driver” event may lead the assessor to make the estimatemore conservative. The A/F should also listen for stereotyping--quickly putting outcomes or information about them intoclasses--which could cause assessors to ignore importantinformation.

Reliance on Internal Coherence.--People often makejudgments ConForm to beliefs and experiences built up throughthe years. They reject new information that appears to beinconsistent with established beliefs or that describes events

they haven1 experienced. The result is that they underestimateuncertainty for unfamiliar events.

The A/F should take care that one outcome does notsound more logical and is not defined more clearly ordescribed in more detail than others. The assessor shouldexamine the uncertainties of subevents to check the plausibilityof the overall outcome. Checklists of assumptions can helpkeep assessors from being bound by their own beliefs.

Reliclnce on Hindsight.--People overestimate theprobability of past events. In hindsight, people find plausibleexplanations for events they did not foresee. Hindsight biasmakes people forget that important information about an eventwas not known before the event occurred (Fischoff and others1982). Hindsight bias causes people to overpredict eventsthey have experienced and underpredict events that have notyet occurred.

The A/F should listen for words such as “inevitable,”“ultimately,” ” should have (known),” and “looking back.”These are clues of hindsight judgment. The A/F should beready to produce contrary evidence, scenarios, and data thatdisagree with past experiences and perhaps challengepreliminary judgments in the assessment process. It may benecessary to withhold information about past outcomes toenable assessors to concentrate on developing a foresightfulperspective.

Group Assessments

It may be desirable or necessary to use a group of expertsto develop a consensus assessment. Several assessors canhave a wider range of knowledge and a greater range ofexperience concerning the uncertain element. Also, theassessment may call for a mixture of key specialties, as inan environmental impact statement_ Biases of some assessorsmay be compensated for by contrasting biases of otherassessors. If the group is skillfully led through the assessmentprocess, group members may gain insights and thus providebetter judgments. Also, assessors may return to theirspecialized fields with a fresh perspective on how theirorganization faces the future and how their work complementsthat of experts in other fields.

Group assessments also have disadvantages. They canbe costly and divert attention from other functions of theorganization. They may also stir up old controversies andarguments between people of different disciplines. Groupassessments do not necessarily provide better estimates ofuncertainty than those provided by a single assessor. Onevery important group bias is “groupthink” (Janis 1972)-&etendency for group members to agree because they want tobe cooperative or because they fear that they will be subjectto mprisals ifthey disagree. This tendency can result in seriousoverconfidence. Groups of assessors from similar disciplinesor interest groups are very prone to groupthink. However,groupthink can be reduced by selecting assessors carefullyand using an astute and experienced A/F.

14

Techniques for developing consensus assessments areof two types: Behavioral approaches ask for initial assessmentsand encourage development of co=nsus through discussion(Seaver 1978). They use group interaction and can be usefulin complex assessments where the element and its outcomeshave no! been defined clearly. They also are useful whenassessing unique or rare events where interaction may becrucial to forming models of causation. Experimentalcomparisons have shown that there is little di&rence betweenresults obtained through the two approaches for well-definedoutcomes and tasks.

There is some evidence that group interaction leadsto groupthink bias when extreme outcomes are estimated.Mathematical approaches combine assessments of respectiveprobabilities by giving more weight to some assessments.There are several options for combining assessmentsmathematically (Winkler 1 WS). However, the arithmetic meanof the individual assessments has been shown to be just asreliable as values obtained by complex systems of weighting.For difftcult assessments involving combinations of effects,differential weighting schemes may be necessary. Weightscan be based on differences in technical ability, the assessors’own ratings of confidence, the importance of the subject area,and the assessors’ records of accuracy in estimating knownuncertainties.

Options for group assessments differ in their degreeof face-to-face interaction (Van & Ven and Delbeog 197 1).One is simply free-form discussion led by the A/F. In the

familiar Delphi technique, assessors remain anonymous toone another. Feedback is coordinated by the A/F, proceed+through several iterations until further improvement is notworthwhile. The nominal group technique consists of silentindividual efforts in a group setting followed by face-to-facediscussion and perhaps voting until a consensus is achieved.There is no best technique, but one highly recommendedhybrid consists ofa Delphi-type process for initial estimatesand one round of revision, followed by structured discussionamong group members, then silent voting on two or threeassessments for each outcome. This estimate-talk-estimatesequence works well in many applications but requires muohtime.

Assessing Rare Events

Some highly unlikely outcomes must be consideredbecause their consequences are so great. Many natural orhuman-caused disasters and human health hazards have thischaracteristic. Sometimes little can be done to influence thelikelihood of a rare outcome, but resources can be allocatedto protect resources, people, or programs at risk.

Probabilities of events or outcomes that might occurless than 1 percent of the time are difficult to assess becausefew people have had any experience with such events oroutcomes. Also, humans natural ly have difftcultydisunguishing among small probability values. Some authors

(Finkell990, von Winterfeldt and Edwards 1986) recommendlogarithms of odds ratios as a useful numerical frame forrare events. The logarithmic scale allows more resolutionand assessor discretion in the tails of distributions.

Selvidge 1980 describes the following procedure. First,have assessors define the outcome in their own words andidentify causes--sets of mutually exclusive events andsequences of events. The sum of the probabilities of theseevents is one. The rare outcome can be modeled using anassessment tree. Second, list the subevents in order ofprobability and estimate the probability of the least likelyevent as a fraction of the probability of the most likely event.Third, compare the likelihood of the most likely event withthat of an event having a probability supported by data orexperience. Fourth, from this base, assign probabilities toother events. Finally, calculate the probability of the rareevent by multiplying across event sequences to find the mostlikely path.

Assessment Quality

Assessment quality is judged on knowledgeability,usefulness, and accuracy. It is possible to evaluate the qualityof the assessment process or the quality of assessment results.We~conductedprooessescangeaeratepoorasse~~ whenknowledge is inadequate. Knowledgeability is an intangiblecriterion, but adherence to procedural rules can ensure thatthe best subjective judgments are obtained.

Usefulness depends both on the output and on theconfidence that decisionmakers have in the assessors’judgment. Decisionmakers are the ultimate judges ofassessment usefulness. Assessments are useful if they aresimple, clear, and in terms that are fitted to the decisionmakers’deliberations. Most people are not used to confrontinguncertainty in numerical or graphic form, so decisionmakersmay have to be trained and gradually introduced to concepts.The A/F can help decisionmakers acquire these new skills.

C&m, accuracy or calibration is not dire&y measurablebecause there is little infonnation on natural occurrences ofthe outcomes. However, the concept of calibration can beused to familiarize assessors with their mission, improveassessors’ methods of gathering information, and improvethe communication of assessments. By assessing theuncertainty of elements for which there is data, the assessorscan develop the skills and attitude required for assessing theuncertainty of elements that are more complex (Winkler andMurphy 1968). Few organizations collect data on outcomesafter assessments have been performed, but feedback fromexperience can improve assessors’ abilities and establishbaseline levels of calibration.

Calibration is a comparison of observed outcomes tothe probabilities assigned by the assessor. An assessor withhigh calibration will would assign a 0.20 probability to anoutcome that data or later experience shows to occur about20 percent of the time. Calibration is important when the

15

Undercontide

essessor

I I I I II 11 1 ’0.0

0.0 0 . 5 1.0

LIKELIHOOD

Figure IO.-Calibration curves fov assessments of likelihods.

number of alternatives in a decision is small, the payoffdifference is quite large (e.g., in life-anddeath emergencies),and when probabilities of several events are linked so thatsystematic errors can be compounded.

A calibration curve is a plot of actual or observedfrequencies against the assessed probabilities for particularoutcomes (fig. 10). The equation of a line representing perfectcalibration would be x = y. Few people, even experts, provideperfectly calibrated assessments; most are overconfident.Overconfident assessors underestimate the probabilities ofinfrequent events and overestimate the probabilities of morecommon events. Ironically, the problem of overconfidenceis worse in difficult assessments. Many people are actuallyunderconfident in simpler assessments.

Calibration is determined by the assessor’s knowledgeof the subject matter and his or her ability to relate thatknowledge to the A/F in probability statements. The assessor’srecognition for scientific achievement or for his or her broadknowledge of a subject does not ensure reliable assessments.Training in probability assessment methods and rapid,frequent, and vivid feedback have been shown to improvethe calibration of assessors.

LITERATURE CITED

Armstrong, Jon Scott. 1978. Long-range forecasting: fromcrystal ball to computer. New York: John Wiley L Sons.612 p.

Bazerman, Max H. 1986. Judgment in managerial decision-making. 2d ed. New York: John Wiley & Sons. 195 p.

Behn, Robert D.; Vaupel, James W. 1982. Quick analysis

for busy decisiomnakers. New York: Basic Books, Inc.415 p.

Beyth-Marom, Ruth; Shlomith, Dekel; Gombo, Routh; Shaked,Moshe. 1985. An elementary approach to thinking underuncertainty. Hillsdale,NJ: Lawrence Erlbaum Association.154 p.

Capen, E.C. 1984. The difficulty of assessing uncertainty.In: Howard, R.A.; Matheson, J.E., eds. Readings on theprinciples and application of decision analysis. 2d ed. MenloPark, CA: Strategic Decisions Group: 591-O.

Cleaves, David A. 1994. Out of uncertainty into forest policy:learning from the FEMAT risk assessments. In: Nof the 1993 annual meeting of the Society of AmericanForesters; November 7-10; Indianapolis, IN. Bethesda, MD:Society of American Foresters: [page numbers unknown].

Cleaves, David A. 1987. Cognitive biases and correctivetechniques: proposals for improving elicitation proceduresfor knowledge-based systems. International Journal of Man-Machine Studies. 27: 155-166.

Cleaves, David A.; Smith, Eric L.; Bradshaw, William G.1987. Subjective assessments in the design and use of forestprotection models. In: Proceedings of the Society ofAmerican Foresters symposium on system analysis inforestry; 1985 December 9-11; Athens, GA. Athens, GA:Georgia Center for Continuing Education, University ofGeorgia. 200 p.

Dawes, Robyn M. 1988. Rational choice in an uncertainworld. Orlando, FL: Harcourt Brace-Jovanovich. 346 p.

Finkel, Adam M. 1990. Confronting uncertainty in riskmanagement: a guide for decisionmakers. Washington, DC:Resources for the Future, Center for Risk Management.68 p.

Fischhoff, Baruch. 1990. Understanding long-term environ-mental risk. Journal of Risk and Uncertainty. 3: 315-330.

Fischhoff, Baruch; Slavic, Paul; Lichtenstein, Sarah. 1982.Lay foibles and expert fables about risk--part 2. TheAmerican Statistician. 36(3): 240-255.

Gregory, Robin; Keeney, Ralph; von Winterfeldt, Detloff.1992. Adapting the environmental impact statement processto inform decisionmakers. Journal of Policy Analysis andManagement. 1 l(1): 58-75.

Ibrekk, H.; Morgan, M.G. 1987. Graphical communicationof uncertain quantities to nontechnical people. Risk Analysis.7(4): 5 19-529.

Janis, I.L. 1972. Victims of groupthink. Boston: HoughtonMiftlin. 250 p.

Johnson, K.N.; Franklin, J.F.; Thomas, J.W.; Gordon J. 1991.Alternatives for management of late-successional forestsof the Pacific Northwest. A report to the Agriculturecommittee and the Merchant Marine Committee of the U.S.House of Representatives. Washington, DC. 59 p.

Kahneman, Daniel; Slavic, Paul; Tversky, Amos, eds. 1982.Judgment under uncertainty: heuristics and biases.Cambridge, UK: Cambridge University Press. 555 p.

Kent, Sherman. 1964. Words of estimated probability. Studiesin Intelligence. 8: 49-64.

16

McNamee, Peti, Celona, John. 1989. Decision analysis withSuper-tree. 2d ed. San Francisco: Scientific Press. 324 p.

Merkhofer, MW. 1987. Quantifying judgmental uncertainty;methodologies, experiences, and insights. Institute ofElectrical and Electronic Engineers (IEEE) Transactionson Systems, Man, and Cybernetics. 17(5): 741-752.

Morgan, M. Granger; Hemion, Max. 1989. Uncertainty: aguide to dealing with uncertainty in quantitative risk 62policy analysis. Cambridge, UK: Cambridge UniversityPress. 325 p.

Saaty, Thomas L. 1988. Decisionmaking for leaders: theanalytical hierarchy process for decisions in a complexworld. Pittsburgh, PA: University of Pittsburgh. 258 p.

Saveland, James M.; Stock, Molly; Cleaves, David A. 1988.Decisionmaking bias in wilderness fire management:implications for expert system development. ArtificialIntelligence (AI) Applications in Natural ResourceManagement. 2(i): 17-29.

Seaver, David A. 1978. Assessing probability with multipleindividuals: group interaction versus mathematicalaggregation. Res. Rep. 78-3. Los Angles, CA: Universityof Southern California, Social Science Research Institute.35 p.

Selvidge, Judith 1980. Assessing the extremes of probabilitydistributions by the fractile method. Decision Sciences. 11:493-502.

Shachter, Robert 1987. Probabilitistic inference andimhXncediagrams. Operations Research. 36: 589604.

Spetzler, Carl S.; Stael Von Holstein, Carl-Axe1 S. 1975.Probability encoding in decision analysis. ManagementScience. 22(3): 340-358.

Talcott, Frederick W. 1992. How certain is that risk estimate?Resources for the Future. 107: 10-15.

Tversky, Amos; Kahneman, Daniel. 1974. Judgement underunce&nty: heuristics and biases. Science. 185: 1124-l 13 1.

Winkler, Robert L. 1968. The consensus of subjectiveprobability distributions. Management Science. 15(2): 61-75.

Winkler, Robert L.; Murphy, Allan L. 1968. Good probabilityassessors. Journal of Applied Meteorology. 7: 751-758.U.S. Department of Agriculture, Forest Service, ForestEcosystem Management Team [Interagency]. 1993. Forestecosystem management: report of the forest ecosystemmanagement assessment team. A report to the Presidentof the United States. Washington, DC. 733 p.

Van de Ven, A.; Delbeog, A.L. 1971. Nominal versusinteracting group processes foroemmittee decisionmakingeffectiveness. Academy of Management Journal. 14(2): 203-212.

von Winterfeldt, Detlof, Edwards, Ward. 1986. Decisionanalysis and behavioral research. Cambridge, UK: Cam-bridge University Press. 604 p.

17 *U.S.GPO:1994-565-016/00010

Cleaves, David A. 1994. Assessing uncertainty in expert judgments about naturalresources. Gen. Tech. Rep. SO-110. New Orleans, LA: U.S. Department ofAgriculture, Forest Service, Southern Forest Experiment Station. 17 p.

Judgmental assessments and predictions of biological, physical, social, andeconomic phenomena are necessary inputs for policy and managerial decisionsabout natural resource management. Uncertainty assessments can be used tocompare alternative projects and risks, help prevent mistakes, communicate andjustify decisions, guide research, and establish ecological monitoring programs.A working de&&ion of uncertainty, a general rationale for quantitying and usinguncertainty in decisionmaking, and a set of structured processes and guidelinesfor estimating uncertain quantities, values, relationships, and events are presented.

Keywords: Decision analysis, judgmental biases, modeling, risk, risk analysis,subjective probability assessment.

The United States Department of Agriculture (USDA) prohibits discriminationin its programs on the basis of race, color, national origin, sex, religion, age,disability, political beliefs, and marital or familial status. (Not all prohibited basesapply to all programs.) Persons with disabilities who require alternative means forcommunication of program information (braille, large print, audiotape, etc.)should contact the USDA Office of Communications at (202) 720-5881 (voice) or(202) 720-7808 (TDD).

To file a complaint, write the Secretary of Agriculture, U.S. Department ofAgriculture, Washington, DC 20250, or call (202) 720-7327 (voice) or (202) 720-1127 (TDD). USDA is an equal employment opportunity employer.

United StatesDepartment of Agriculture

Forest Service

Southern ForestExperiment Station701 Loyola Ave., Rm.T-10210New Orleans, LA 70113-1931

OFFICIAL BUSINESSPenalty for Private Use $300