The Value of Information for Managing Contaminated SedimentsMatthew E. Bates,*,† Magnus Sparrevik,‡,§,∥ Nicolas de Lichy,⊥ and Igor Linkov†

†Environmental Laboratory, Engineer Research and Development Center, US Army Corps of Engineers, 696 Virginia Road, Concord,Massachusetts 01742, United States‡The Norwegian Defence Estates Agency, Forsvarsbygg, P.O. Box 405 Sentrum, Oslo, NO-0103, Norway§Norwegian Geotechnical Institute, P.O. Box 3930 Ulleval Stadion, Oslo, NO-0806, Norway∥Department of Industrial Economics and Technology Management, Norwegian University of Technology, Trondheim 7491,Norway⊥London School of Economics, Houghton Street, London WC2A 2AE, United Kingdom

*S Supporting Information

ABSTRACT: Effective management of contaminated sediments is important for long-term human andenvironmental health, but site-management decisions are often made under high uncertainty and withoutthe help of structured decision support tools. Potential trade-offs between remedial costs, environmentaleffects, human health risks, and societal benefits, as well as fundamental differences in stakeholder priorities,complicate decision making. Formal decision-analytic tools such as multicriteria decision analysis (MCDA)move beyond ad hoc decision support to quantitatively and holistically rank management alternatives andadd transparency and replicability to the evaluation process. However, even the best decisions made underuncertainty may be found suboptimal in hindsight, once additional scientific, social, economic, or otherdetails become known. Value of information (VoI) analysis extends MCDA by systematically evaluating theimpact of uncertainty on a decision. VoI prioritizes future research in terms of expected decision relevanceby helping decision makers estimate the likelihood that additional information will improve decisionconfidence or change their selection of a management plan. In this study, VoI analysis evaluatesuncertainty, estimates decision confidence, and prioritizes research to inform selection of a sedimentcapping strategy for the dibenzo-p-dioxin and -furan contaminated Grenland fjord system in southern Norway. The VoI modelextends stochastic MCDA to model decisions with and without simulated new information and compares decision confidenceacross scenarios with different degrees of remaining uncertainty. Results highlight opportunities for decision makers to benefitfrom additional information by anticipating the improved decision confidence (or lack thereof) expected from reducinguncertainties for each criterion or combination of criteria. This case study demonstrates the usefulness of VoI analysis forenvironmental decisions by predicting when decisions can be made confidently, for prioritizing areas of research to pursue toimprove decision confidence, and for differentiating between decision-relevant and decision-irrelevant differences in evaluationperspectives, all of which help guide meaningful deliberation toward effective consensus solutions.

■ INTRODUCTION

Over the past centuries, harmful contaminants from urban andindustrial effluent, agricultural runoff, and other point andnonpoint sources have accumulated in sediments around theworld, causing significant health risks to humans and ecologicalreceptors. Managing contaminated sites for the protection ofhuman and environmental health has become a common issuewith significant implications for a variety of stakeholdersincluding local residents, regulators, and commercial andindustrial land users. The multitude of stakeholder perspectives,combined with considerable uncertainties about the impacts ofcontamination and the effects of various remedial alternatives,make managing contaminated sediments highly complex.1−6

Uncertain information can result in significant cost associatedwith management.7 Current methods for management includevarious forms and extents of capping, removal, and activetreatment.8 These can be evaluated against each otheraccording to their expected costs, benefits, and risks.9,10 Asany sediment management alternative is unlikely to simulta-

neously have the largest benefit, lowest cost, and least risk,decision makers must conduct trade-off analyses to determinewhich alternative will provide the greatest holistic value.Traditional decision support methods often fail to effectively

evaluate all important aspects of complex problems that may bebetter suited for analyses leveraging systematic and quantitativedecision-analytic methods.11−13 Methods such as multicriteriadecision analysis (MCDA) allow decision makers to separatelycapture each important aspect of the problem, describe trade-offs and priorities between objectives, incorporate uncertainties,rank alternatives, and transparently share results with thestakeholder community.14−16 These tools are valuable becausethey decompose complex problems into more tractable parts,allow one to ask the right stakeholders or experts for each

Received: February 11, 2014Revised: June 19, 2014Accepted: June 24, 2014

Article

pubs.acs.org/est

© XXXX American Chemical Society A dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXX

different piece of information, and provide transparency andreplicability to the process. These methods can morethoroughly evaluate disparate remedial alternatives since theyallow decision makers to simultaneously consider multipledecision criteria and clearly communicate why alternatives areranked as they are. In recent years, this type of analysis has beenincreasingly applied to complex environmental problems fromcontaminated sediment management to ecosystem restorationand spatial planning.10−11,17−20

Even properly structured remediation decisions are oftenbased on uncertain data, which adds further complexity. Factorssuch as imprecise knowledge about how managementalternatives will ultimately perform in terms of each criterion,a need for trade-offs between competing criteria, anddifferences in priorities between evaluation perspectives, makeit difficult to evaluate alternatives with full and transparentconsideration of all relevant detail. Stochastic MCDA methodscan account for uncertain inputs by using Monte Carlosimulations to produce a probability distribution of results,21,22

but even these distributions of alternative rankings may changeas new information is gained. To determine how differentreductions in uncertainty may affect decision confidence andalternative rankings, analysts can perform a value of information(VoI) analysis.23−25 This estimates changes in decisionoutcomes that could result from gaining specific informationrelated to MCDA inputs prior to making a decision. Similar to adecision-focused sensitivity analysis, the results show whichdata is most important to the decision, help direct researchefforts toward decision-relevant information, and help decisionmakers identify the minimum research needed to confidentlychoose a management alternative.26−28 The importance ofquantitatively prioritizing research and estimating decisionconfidence has been explicitly emphasized by recent US andinternational initiatives,11,17,29 and lack of robust decisionconfidence indicators has been a criticism of traditional MCDAmodeling.30−32 In comparison to the resources required forphysical research, the resources required to structure a remedialdecision through MCDA/VoI analysis are minor, but thepotential benefits of MCDA/VoI modeling can be vast.In this paper, VoI is applied to a contaminated-sediment

management case study in the dibenzo-p-dioxin and -furan(PCDD/F) contaminated Grenland fjord system of southernNorway. This VoI builds on a stochastic MCDA performed forthe Grenland fjord site by Sparrevik, Barton, Bates, andLinkov33 (SBBL), a previous paper by the authors. That paperuses stochastic MCDA to rank remedial alternatives based onestimated performance across human health, societal benefit,remedial cost, and environmental impact criteria. This is donefrom cost effectiveness, cost benefit, and value plural evaluationperspectives and incorporates uncertainty related to alternativeperformance. The resulting distributions of alternative rankingsindicate that no alternative is clearly dominant given thecurrently available information and that further action may beneeded to reduce uncertainty prior to making a selection. Thispaper extends the previous stochastic MCDA for Grenlandfjord with a decision-directed VoI analysis that estimatesdecision confidence under different information-gatheringscenarios, prioritizes research strategies, and projects the degreeto which different types of information are expected to changewhich alternatives are perceived as best.

■ METHODSCase Study MCDA Problem Formulation. Sediments in

the 66 km2 Grenland fjord system in southern Norway (Figure1) are contaminated with PCDD/Fs from industrial activity by

the Hydro Porsgrunn magnesium plant that has beendischarging wastewater to the area since 1951, and withpolychlorinated biphenyls (PCBs) from other local sources.Reductions in toxic discharges have measurably decreasedconcentrations in marine organisms but levels are still high,leading to fishing restrictions and health advisories againstconsuming local seafood.34 Mitigating measures, includingisolating the contaminated sediments by covering them with athin layer cap of clean material, are currently being proposedand studied.35,36

SBBL formulate the sediment capping decision for theGrenland fjord as an MCDA problem consisting of thefollowing steps: (1) formulating general objectives, (2)enumerating alternatives management plans, (3) developingspecific criteria and metrics that decompose the generalobjectives into measurable attributes, (4) measuring theperformance of each alternative remediation plan againstthose criteria, and (5) weighting the relative importance ofeach criterion with respect to the others before calculatingresults. SBBL use MCDA to rank six remedial alternatives forthe fjord. These include a low-cost, minimally effective optionof natural recovery (NR), in which natural resedimentationcovers contaminated sediments over time, and also a high-cost,highly effective option for whole fjord capping (WFC) thatincludes contaminated sediments in the entire inner fjord and

Figure 1. Grenland fjord system in southern Norway, showing theextent of six remedial alternatives under consideration in the MCDA &VoI analyses. The inner and outer fjord systems are separated by theshallow Brevik Sill. Remedial alternatives include natural recovery(NR; green, yellow, red, and orange), capping only the highestcontaminated areas in the inner fjord (HIFC; orange) or outer fjord(HOFC; yellow), capping the entire area of the inner fjord (IFC; redand orange), or a larger extent of the outer fjord (OFC; green andyellow), and capping the whole inner fjord plus a large portion of theouter fjord (WFC; green, yellow, red and orange). Figure reprintedfrom ref 33. Copyright 2012 American Chemical Society.

Environmental Science & Technology Article

dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXXB

most of the outer fjord. Intermediate alternatives includeoptions for inner fjord capping (IFC), outer fjord capping(OFC), or capping of only the highly contaminated hot spotsin the inner (HIFC) or outer (HOFC) fjord.Four criteria are used to compare and evaluate the six

alternative capping approaches: human health risk reduction,socio-economic benefits expected from removing seafoodconsumption advisories, remediation cost, and estimated lifecycle environmental impacts. Because the performance of thesix alternatives on each of these four criteria is uncertain,skewed normal distributions of performance scores are usedbased on input data for median, 5th percentile, and 95thpercentile values for each alternative for each criterion. Thesedistributions represent the best a priori expert interpretation ofliterature estimates of potential remediation outcomes in thefjord, as shown graphically in Supporting Information, Table S1and as explained by SBBL. Trade-offs between the four criteriaare applied differently in each of three weighting schemesmeant to replicate cost effectiveness (CE), cost benefit (CB),and value plural (VP) methods of evaluation. The costeffectiveness weighting scheme gives equal weight to healthrisk reductions and costs but no weight to the other criteria.The cost benefit weighting scheme gives equal weight to socio-economic benefits and costs but no weight to the others. Thevalue plural weighting scheme gives equal weight to all fourcriteria. (Further detail regarding the case study alternatives,criteria, weighting schemes, and performance data are describedby SBBL; see also Supporting Information.)PROMETHEE II MCDA. Once the decision problem is

formulated, different MCDA methods can be applied withslightly different assumptions and operations to normalize andaggregate raw performance score data across criteria to derive atotal relative preference score for each alternative. AvailableMCDA methods can generally be categories as either multi-attribute utility theory (MAUT) or outranking methods.MAUT methods use specific knowledge of utility or valuefunctions to infer preference for different levels of alternativeperformance across a fixed scale, whereas outranking methodsavoid assumptions about the scales used to normalize higher orlower performance scores into preference values and insteaduse pairwise comparisons to identify the relative dominanceamong each pair of alternatives and aggregate those scores intototal preference scores.16 For cases where the input data isuncertain, stochastic MCDA can use Monte Carlo simulationsto repeatedly apply the chosen MCDA formula to each set ofalternative performance data sampled from the inputdistributions.SBBL use the PROMETHEE II outranking MCDA

method37 in 10 000 Monte Carlo simulations to estimate thedistribution of total preference scores (“net flows”) for eachmanagement alternative (i.e., natural recovery or some extent ofcapping) under each weighting scheme. In each simulation, apreference function, P(k,a), compares the performance score ofalternative k directly with the performance scores of otheralternatives, for example, a, on some criterion to establishrelative dominance on a 0−1 scale where a score of 0 representsthe strict domination of k by a and a score of 1 represents thestrict dominance of k over a.An aggregate dominance score, φj′(k), is calculated for each

alternative on each criterion based on preference functionsevaluating each alternative k with all other alternatives, where jrepresents the decision criterion (i.e., health risk, socio-economic benefit, life cycle environmental impact, or cost), k

denotes the alternative under consideration, A represents theset of all alternatives, and v represents the number ofalternatives to be evaluated (valid for v > 1):

φ′ =∑ −

−≠ ∈k

P k a P a k

v( )

( ( , ) ( , ))

1ja k A j j

The value of φj′(k) is constant across weighting schemes andranges from −1 to 1 for each alternative on each criterion,where a score of +1 implies that alternative k is the strictly mostpreferred among all alternatives on criterion j, and a score of −1implies that alternative k is the strictly least preferred among allalternatives on criterion j. Here, φj′(k) uses a linear comparisonfunction with a threshold of strict preference, p, set at 10% ofthe range (maximum to minimum) of values for each criterion(see Supporting Information and Figure S1 for additionaldetails on the threshold of strict preference and its use).As the dominance scores for each alternative are independent

of the weighting scheme (i.e., CE, CB, or VP), i, weighing isapplied to aggregate flows as a weighted sum across criteria tocalculate the net flow, φj(k), which defines the final rank orderand total relative dominance of each alternative for eachweighting scheme:

∑φ φ= · ′k w k( ) ( )ij

ij j

where the weight, wij, indicates the relative importance ofcriterion j among all other criteria (out of 100% total weight)under weighting scheme i. Since net flow is an expression of therelative degree to which each alternative outperforms theothers, it can be interpreted as a measure of decision conf idence,where higher values on a 0−1 scale represent greaterconfidence in the dominant performance of an alternative onthe criteria that matter most.

PROMETHEE II Expected Value of Perfect and Partial-Perfect Information. When summarizing the results of astochastic MCDA, the expected alternative ranking remainsuncertain because the knowledge of underlying input perform-ance scores are uncertain. However, not all uncertainties in theinput data are equally relevant to the decision becauseuncertainties in performance scores contribute differently toresulting net flow and rank order. (For example, it is possiblefor uncertainty on the least important criterion to hold thegreatest sway over the decision if there is great overlap amongalternatives on that criterion but no overlap among othercriteria and otherwise similar net flow scores.) Through VoIanalysis, it is possible to determine which data uncertainties areexpected to have the greatest average effect on the total score ofeach alternative, potentially enabling selection of a betteralternative than would be selected in the base case. Decision-directed VoI analysis is seen as an integral part of environ-mental decision making, where alternatives are evaluated,decision confidence is assessed, the minimum necessaryresearch is defined and prioritized, science is undertaken untilprioritized uncertainties are reduced, and site managementaction is ultimately taken (Figure 2).Howard23 and Raiffa24 define the value of additional

information as the difference between the expected value ofthe outcome that would be achieved under current knowledgeand the expected value that could be achieved under posteriorknowledge. In classical approaches, this value is representedeconomically or in terms of utility. Consistent with the SBBLPROMETHEE II model formulation, this study adapts classical

Environmental Science & Technology Article

dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXXC

approaches following Linkov et al.26 to represent this value interms of increased net flow, the holistic total score that resultsfrom evaluating each alternative against the weighted criteria ofthe MCDA model. Since net flow is a weighted measure of theextent to which one alternative outperforms all others in theareas that are most important, VoI in this context is notinterpreted as a monetary value but rather as an improvementin decision confidence (e.g., related to discovering that analternative outperforms the others even more strongly thaninitially believed) due to the additional information.The following VoI model extends the SBBL stochastic

MCDA into a two-stage nested Monte Carlo simulation ofdecisions with and without additional information to infer thepotential impact of information on decision outcomes, that is,to identify which uncertainties are most decision-relevant. Themodel first simulates gaining perfect information about one ormore uncertain input variables by probabilistically samplingvalues from the skewed normal input distributions. Eachsampled value represents the true performance of somealternative in some trial, corresponding to one potential truefuture state of the world. While different in each trial, across thetrials the observed distribution of sampled points will reflect theinput distribution from which it is drawn.Then, within each trial, a second Monte Carlo simulation

runs to sample values for the remaining variables for whichperfect knowledge is not simulated and calculates a distributionof expected net flows given the perfectly known informationpresumed in the first trial. While it is impossible to know apriori what the actual future state of the world will be, acomparison of average net flows from simulations with someperfect information to those without information in the basecase sets a bound on the expected value of partial perfectinformation (EVPPI) from that type of research. We interpretthis an upper bound because we expect most real worldresearch projects to provide something less than perfectinformation. To find the expected value of perfect information(EVPI), the upper bound on the value of all research on thistopic, information on every uncertainty is simulated assimultaneously known and available before the decision.“Partial perfect” information assumes absolute certainty on

one or more input values without addressing the uncertainty ofthe others, while complete perfect information assumessimultaneous certainty of all inputs is available in each trial.In these VoI simulations, any increase in average net flow overthe base case can be attributed to simulations where analternative perceived as inferior on average in the base case isactually seen to outperform all other alternatives once specificinformation is simulated, enabling selection of a higher scoring

alternative in those simulations than in the base case. On thebasis of the outcome of the VoI analysis, research can betargeted to reduce uncertainty on input values where thepayoffin this case improved total net flow scores fromselecting the best actual management alternativeis largest.In the base case where no additional information is

generated, the expected net flow of the optimal decision (i.e.,to choose the alternative with the maximum average net flow)is calculated for each weighting scheme as

∑φ φ=_=

k Nmax ( ( ))/i kn

N

i nno1

,

where N is the number of simulations, here implemented as N= 10 000. This represents selection of the alternative (k) fromthe stochastic MCDA with the highest average score based oncurrent uncertain estimates of future performance.It is often possible to improve decision confidence by

collecting additional information prior to making a selection.An upper bound for the value of engaging in research isassessed by simulating perfect knowledge of all uncertainties,selecting the highest scoring alternative in each trial, and thentaking the average of those various highest scores. While we donot know which scores will ultimately emerge in the real world,this simulation shows how high a score we could expect onaverage if we knew all relevant information prior to needing tochoose an alternative. Because the decision maker has perfectinformation prior to each decision in each trial, the expectednet flow with perfect information is equivalent to the average ofthe net flows of the highest scoring alternative in each of N =10 000 simulations. Average net flow with perfect information,φperfect, is calculated for each weighting scheme as

∑φ φ=_=

N(max ( (k)))/in

N

i nperfect1

k ,

The difference between the simulated net flow scores with andwithout information (the EVPI) can reveal the extent to whichsufficient information is already present to make a reasonabledecision: EVPIi = φi_perfect − φi_no. If this difference between netflows with and without new information is minor, there may belittle to be gained by waiting (or paying) for additionalinformation to be revealed prior to selecting an alternative.It is also possible to quantify relative contributions to

increased net flow from information about only a subset ofcriteria, which is useful for prioritizing research between criteria.For example, given two research efforts of otherwise similarscope and cost but investigating alternative performance ondifferent criteria, is one expected to change our base casedecision more than the other? For criteria in each subset C forwhich information is simulated as known, the aggregatedominance score, F, for each alternative is calculated in eachof N = 10 000 trials as Fj(k) = φj′(k), as defined previously. Theknown scores of criteria in C for which we are simulatingperfect information are sampled once in each of the N trials andthen presumed known for each of M = 10 000 additional,second-stage inner simulations of other criteria not beingresearched in C. For criteria not in subset C, an alternativemeans of calculating the aggregate dominance score takes anaverage over M inner-stage Monte Carlo simulations: Fj(k) =Σm = 1M (φj,m′ (k))/M. On the basis of these two means of

calculating Fj(k) depending on whether or not each j is in C,the total net flow for each alternative for each weighting scheme

Figure 2. Schematic expression of the integral role of stochasticMCDA and VoI analyses in environmental decision-making underuncertainty.

Environmental Science & Technology Article

dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXXD

in each of the N simulations is calculated as the weighted sum:φi(k) = Σjwij·Fj(k). Finally, these values are averaged over all Ntrials to obtain the expected net flow of the optimal decisionwith perfect information about only criteria in C and whileretaining the original uncertainty of all other criteria not in C:

∑φ =_=

F k N(max ( ( )))/i Cn

N

k i1

The expected increase in net flow resulting from perfectinformation about only the criteria in C is calculated as EVPPIi= φi_c − φi_no. This is done for all possible combinations ofcriteria C to compare improvements in decision confidenceexpected from different research strategies. This is helpful forprioritizing research strategies based on their decisionrelevance. The value of information simulated under variousresearch portfolios can be compared with the monetary andtemporal costs required to gather reasonably completeinformation in those portfolios, to prioritize research givenlimited resources.

■ RESULTS AND DISCUSSIONStochastic MCDA Results. Results of the SBBL stochastic

MCDA (Supporting Information, Figures S2 and S3) indicatedthat partial remediation of the inner fjord (HIFC) was thepreferred alternative under the value plural weighting scheme.HIFC was also most preferred under the cost effectivenessweighting scheme, though the differences in net flow under thisscheme were small and all alternatives scored close to zero(indicating near equivalence). Under the cost benefit weightingscheme, partial remediation of the outer fjord (HOFC) wasidentified as the most preferred alternative. Similar results arereproduced in this paper, using a threshold of strict preferenceof 10%, and extended into VoI results.Rank order results (Figure 3) give decision makers an

indicator of decision confidence by showing how consistently

each alternative can be expected to be ranked in a certain orderrelative to the other alternatives, under existing uncertaintyregarding decision model inputs. Although the stochasticMCDA indicates the average preferred alternative from a costeffectiveness perspective to be HIFC, when perfect informationis available, HIFC is seen to rank first only about 40% of thetime, with other alternatives outranking it nearly 60% of thetime. Similarly, the average preferred alternatives from the costbenefit and value plural evaluation perspectives, HOFC andHIFC respectively, are expected to rank first in only about half

of the simulations where perfect information is present.Consequently, the alternatives expected to be selected on thebasis of the stochastic MCDA analysis have a high chance ofbeing found suboptimal once further information is revealed.These results could be an indicator for many decision makersthat further data collection to reduce uncertainties prior tomaking a decision is worth pursuing.

Value of Information Results. Funding, equipment, andstaffing limitations often mean that it is only feasible to pursuelimited research prior to action. VoI results can help prioritizefurther Grenland fjord research in terms of expected decisionimpact. The net flow values with no additional informationrelate to the level of decision confidence that a decision makermight have regarding a chosen alternative performing betterthan the other alternatives in the base case. Similarly, theaverage flow net value with perfect information indicates themaximum decision confidence that can be expected if allpossible research were performed prior to the decision. Theabsolute and percent differences in these values help decisionmakers decide if additional research may be warranted.Results show the average net flow of the highest scoring

alternatives under different simulated information scenarios(Figure 4). From a cost effectiveness perspective, perfectinformation on both health risks and costs (the weightedcriteria in that scheme) increases the average net flow by 0.16points (179% increase), and partial perfect information on costsalone increases net flow by 0.10 points (a 115% increase) or onhealth risks alone increases net flow by 0.09 points (a 98%increase). From a cost benefit perspective, perfect informationon both societal benefits and costs (the weighted criteria in thatscheme) increases the average net flow by 0.17 points (a 52%increase), and partial perfect information on societal benefitsalone increases net flow by 0.13 points (a 41% increase) or oncosts alone increases net flow by 0.04 points (an 11% increase).From a value plural perspective, research into environmental

risks (omitted from Figure 4) is expected to be of negligiblerelevance, increasing net flow by less than 0.01 points over anygiven research portfolio without it. Research into societalbenefits is expect to be most useful, increasing net flow by 0.06points (a 30% increase), followed by research into health risksalone and then costs alone, in that each are expected to increasenet flow by about 0.10 points (a 5% increase). Research intomultiple criteria simultaneously can further increase net flowover the gains from researching single criteria, but with smalleradditional percent gains. Perfect information on all four criteriais simulated as increasing average net flow by 0.08 points (a43% increase). Note that this evaluation perspective weights allcriteria equally but shows that even perfect information onenvironmental effect is anticipated to have negligible effect onthe decision. This shows that while environmental factors arestill equally important to the decision, they are alreadysufficiently well quantified to enable robust decision makingand further research in to this topic should not be prioritizedover research in more decision-relevant areas.Without information, the decision maker is simulated as

making one choice, selecting the alternative with the highestaverage net flow (as identified in Figure 3 and the leftmostchart of each panel in Figure 5). Knowledge of partial-perfectinformation both increases the average net flow of thesimulated decision (Figure 4) and changes the ratios in whichthe alternatives are chosen in the simulations (remaining chartsof each panel in Figure 5). The charts in Figure 5 show theratios of how often each alternative is chosen in each of the

Figure 3. Percent of trials in which each alternative is ranked first overthe 10 000 Monte Carlo simulations of the MCDA/VoI modelsimulating perfect information under each weighting scheme.

Environmental Science & Technology Article

dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXXE

information scenarios summarized by the bars in Figure 4.Using this to compare perceived preference between thealternatives with different information provides decision makersanother visual tool to reference when deciding how muchinformation to pursue (e.g., for identifying which partial perfectinformation scenario seems sufficiently close to the perfectinformation scenario that additional refinement may not benecessary). For example, from the cost benefit perspective (topright panel of Figure 5), the spread of perceived preferenceamong the alternatives in the “Societal info” chart lookssufficiently similar to that of the “Societal & Cost Info” chartthat a decision maker might forego pursuing additional costresearch even though Figure 4 shows that it could still increaseaverage net flow.Application to Environmental Site Management.

Complex environmental decisions are often made withuncertain data and while facing trade-offs between multiple

and often conflicting criteria. MCDA has advantages overlower-dimensional decision methods because it can trans-parently incorporate stakeholder and management prioritiesbetween the different criteria, simulate multiple evaluationperspectives, and integrate data measured on different scales toholistically evaluate and rank all management alternatives underconsideration. Stochastic MCDA extends traditional MCDA byacknowledging that alternative rankings cannot be establishedwith absolute certainty due to a frequent lack of knowledgeabout the eventual performance of the alternatives on thevarious criteria. Stochastic MCDA describes alternativeperformance through probabilistic distributions that reflectthese uncertainties, which helps decision makers make the bestuse of current information to identify alternatives expected tohave the most preferable average outcomes.However, when performance scores are highly uncertain, as is

common in many environmental applications, resulting

Figure 4. Average total net flows of the top ranking alternatives in each trial for each weighting scheme: with no additional research, with partialperfect information on each individual criterion, with partial perfect information on combinations of criteria, and with perfect information on allcriteria.

Figure 5. Proportion of time each alternative is chosen in each information scenario from each evaluation perspective. Ratios of alternatives thatdraw closest to ratios under perfect information are desired. (Each pie chart decomposes a net flow bar from Figure 4 to show how each alternative isrepresented in the average net flow score in each scenario.)

Environmental Science & Technology Article

dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXXF

decision confidence can remain low and the expected rankorder from a stochastic MCDA may be subject to change asnew information affecting the input distributions is introduced.When faced with low decision confidence, decision makers andstakeholders may be tempted to try to reduce uncertainties inthe analysis through unguided collection of additional data. Asdemonstrated in this paper, VoI analysis can extend stochasticMCDA by simulating various potential information gatheringstrategies for reducing the underlying uncertainties. By showinghow rankings and net flows are expected to change underdifferent information scenarios, decision makers and research-ers can differentiate between decision-relevant and decision-irrelevant uncertainties, determine if existing information issufficient for decision making, and design research plansreasonably expected to most improve decision making with theleast additional investment (Figure 2).This choice of which research strategy to pursue should

reflect consideration of the time and resources available for use,the decision confidence expected in the base case (Figure 3),the expected absolute net flows with and without newinformation (Figure 4), the expected percent increase in netflow available through new information (Figure 4), and therelative agreement in perceived rank order simulated in caseswith partial perfect vs full perfect information (Figure 5). Whenconsidered together, these results are informative for movingbeyond qualitative or intuitive treatments of uncertainty toquantify information value. Environmental site managersprovided with these types of transparent and quantitativeresults can better choose which research areas to invest in, canquickly run sensitive and scenario analyses regardingdistribution shape, modeled criteria, weighting and evaluationperspectives, etc., and can more clearly communicate thepotential impact of further research to partners and stake-holders.The Grenland Fjord results both illustrate the importance of

selecting an appropriate evaluation perspective (and ofcomparing results across perspectives) and highlight thepotential differences in decision relevance of different research.This emphasizes the importance of prioritizing research basedon expected outcome in the specific problem context ratherthan based on researcher familiarity or some other qualitativemeasure. VoI results can provide a transparent and objectivebasis for furthering discussions between stakeholders anddecision makers, quantifying the potential impact of research(or lack thereof), prioritizing research based on expectedoutcomes, and clearly showing how uncertainties in input dataare expected to affect decision confidence and the ranking ofalternative site management plans. As an analytical effort, thetime and resources needed for a VoI analysis are ofteninsignificant in comparison to the resources they can save byprioritizing and bounding the need for further physical research.

■ ASSOCIATED CONTENT*S Supporting InformationMore detailed information is available about the preferencefunction used in the PROMETHEE II Outranking MCDAmethod, background on the Grenland fjord region, and theinput data and results from the SBLL model. This material isavailable free of charge via the Internet at http://pubs.acs.org.

■ AUTHOR INFORMATIONCorresponding Author*E-mail: Matthew.E.Bates@usace.army.mil.

NotesThe authors declare no competing financial interest.

■ ACKNOWLEDGMENTS

The authors would like to thank previous studies for the workperformed on the Norwegian Grenland fjord system, especiallythe Opticap project (www.opticap.no), the SEDFLEX project,and the Norwegian Research Council that financed thosestudies. Thanks are also due to Kelsie Baker and Cate Fox-Lentfor help with graphics, editing, and data curation, and toProfessor Jeff Keisler who was instrumental in teaching andcritiquing the VoI methods. The authors additionally thank Dr.Todd Bridges, Dr. Martin Schultz, and are grateful for financialsupport from the Dredging Operation Environmental Research(DOER) program of the US Army Corps of Engineers(USACE). Permission was granted by the USACE Chief ofEngineers to publish this material.

■ REFERENCES(1) Bridges, T. S.; Apitz, S. E.; Evison, L.; Keckler, K.; Logan, M.;Nadeau, S.; Wenning, R. J. Risk-based decision making to managecontaminated sediments. Integr. Environ. Assess. Manage. 2006, 2 (1),51−58.(2) Forstner, U.; Salomons, W. Trends and challenges in sedimentresearch 2008: The role of sediments in river basin management. J.Soils Sediments. 2008, 8 (5), 281−283.(3) Weber, R.; Gaus, C.; Tysklind, M.; Johnston, P.; Forter, M.;Hollert, H.; Heinisch, E.; Holoubek, I.; Lloyd-Smith, M.; Masunaga, S.;Moccarelli, P.; Santillo, D.; Seike, N.; Symons, R.; Torres, J. P. M.;Verta, M.; Varbelow, G.; Vijgen, J.; Watson, A.; Costner, P.; Woelz, J.;Wycisk, P.; Zennegg, M. Dioxin-and POP-contaminated sitesContemporary and future relevance and challenges. Environ. Sci.Pollut. Res. 2008, 15 (5), 363−393.(4) Sparrevik, M.; Breedveld, G. D. From ecological risk assessmentsto risk governance. evaluation of the Norwegian management systemfor contaminated sediments. Integr. Environ. Assess. Manage. 2010, 6(2), 240−248.(5) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Seager, T. P.; Bridges, T.;Gardner, K. H.; Rogers, S. H.; Belluck, D. A.; Meyer, A. Multicriteriadecision analysis: a comprehensive decision approach for managementof contaminated sediments. Risk Anal. 2006, 26 (1), 61−78.(6) National Research Council. Contaminated Sediments in Ports andWaterways: Cleanup Strategies and Technologies; National AcademiesPress: Washington, DC, 1997.(7) Linkov, I.; Ames, M. R.; Crouch, E. A.; Satterstrom, F. K.Uncertainty in octanol−water partition coefficient: Implications forrisk assessment and remedial costs. Environ. Sci. Technol. 2005, 39(18), 6917−6922.(8) Ghosh, U.; Luthy, R. G.; Cornelissen, G.; Werner, D.; Menzie, C.A. In-situ sorbent amendments: A new direction in contaminatedsediment management. Environ. Sci. Technol. 2011, 45 (4), 1163−1168.(9) Linkov, I.; Seager, T. P. Coupling multi-criteria decision analysis,life-cycle assessment, and risk assessment for emerging threats.Environ. Sci. Technol. 2011, 45 (12), 5068−74.(10) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Seager, T. P.; Bridges,T.; Gardner, K. H.; Rogers, S. H.; Belluck, D. A.; Meyer, A.Multicriteria decision analysis: A comprehensive decision approach formanagement of contaminated sediments. Risk Anal. 2006, 26 (1), 61−78.(11) Gamper, C. D.; Turcanu, C. On the governmental use of multi-criteria analysis. Ecol. Econ. 2007, 62 (2), 298−307.(12) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Batchelor, C.; Bridges,T.; Ferguson, E. From comparative risk assessment to multi-criteriadecision analysis and adaptive management: Recent developments andapplications. Environ. Int. 2006, 32 (8), 1072−1093.

Environmental Science & Technology Article

dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXXG

(13) Linkov, I.; Moberg, E. Multi-Criteria Decision Analysis:Environmental Applications and Case Studies; CRC Press: BocaRaton, FL, 2012.(14) Howard, R. A. The foundations of decision analysis. IEEE Trans.Syst. Sci. Cyber., 1968, 4 (3), 211−219.(15) Keeney, R. L.; Raiffa, H. Decisions with Multiple Objectives:Preferences and Value Tradeoffs; Wiley: New York, 1976.(16) Belton, V.; Stewart, T. J. Multiple Criteria Decision Analysis anIntegrated Approach; Kluwer Academic: Boston MA, 2002.(17) National Research Council. Environmental Decisions in the Faceof Uncertainty; The National Academies Press: Washington, DC, 2013.(18) Mendoza, G. A.; Martins, H. Multi-criteria decision analysis innatural resource management: A critical review of methods and newmodelling paradigms. Forest Ecol. Manage. 2006, 230 (1−3), 1−22.(19) Alvarez-Guerra, M.; Canis, L.; Voulvoulis, N.; Viguri, J. R.;Linkov, I. Prioritization of sediment management alternatives usingstochastic multicriteria acceptability analysis. Sci. Total Environ. 2010,408 (20), 4354−4367.(20) Linkov, I.; Sahay, S.; Kiker, G.; Bridges, T.; Seager, T. P. Multi-criteria decision analysis: A framework for managing contaminatedsediments. In Strategic Management of Marine Ecosystems; Springer:Amsterdam, The Netherlands, 2005; pp 271−297.(21) Hyde, K.; Maier, H. R.; Colby, C. Incorporating uncertainty inthe PROMETHEE MCDA method. J. Multi-Criteria Decis. Anal. 2003,12 (4−5), 245−259.(22) Tervonen, T.; Figueira, J. R.; Lahdelma, R.; Dias, J. A.; Salminen,P. A stochastic method for robustness analysis in sorting problems.Eur. J. Operat. Res. 2009, 192 (1), 236−242.(23) Howard, R. A. Information value theory. IEEE Trans. Syst. Sci.Cybernetics, 1966, 2 (1), 22−26.(24) Raiffa, H. Decision Analysis: Introductory Lectures on Choices underUncertainty; Addison-Wesley: Reading MA, 1968.(25) Keisler, J. M.; Collier, Z. A.; Chu, E.; Sinatra, N.; Linkov, I.Value of information analysis: The state of application. Environ. Syst.Decis. 2013, DOI: 10.1007/s10669-013-9439-4.(26) Linkov, I.; Bates, M. E.; Canis, L. J.; Seager, T. P.; Keisler, J. M.A decision-directed approach for prioritizing research into the impactof nanomaterials on the environment and human health. Nat.Nanotechnol. 2011, 6 (12), 784−787.(27) Brennan, A.; Kharroubi, S.; O’Hagan, A.; Chilcott, J. Calculatingpartial expected value of perfect information via Monte Carlo samplingalgorithms. Med. Decis. Making 2007, 27 (4), 448−470.(28) Keilser, J. M. Value of information: Facilitating targetedinformation acquisition in decision processes. Environ. Systems andDecisions 2014, DOI: 10.1007/s10669-014-9493-6.(29) The Presidential/Congressional Commission on Risk Assess-ment and Risk Management. Framework for Environmental Health RiskManagement, Final Report, Volume 1. Washington, DC, 1997; http://oaspub.epa.gov/eims/eimscomm.getfile?p_download_id=36372.(30) Hubbard, D.; Samuelson, D. Modeling without Measurements:How the decision analysis culture’s lack of empiricism reduces itseffectiveness. OR/MS Today 2009, 36 (5), 26−31, http://www.orms-today.org/orms-10-09/frrisk.html.(31) Morgan, M. G.; Henrion, M. Uncertainty: A Guide to Dealingwith Uncertainty in Quantitative Risk and Policy Analysis; CambridgeUniversity Press: New York, 1990.(32) Felli, J. C.; Hazen, G. B. Do sensitivity analyses really captureproblem sensitivity? An empirical analysis based on information value.Risk, Decis. Policy 1999, 4 (2), 79−98.(33) Sparrevik, M.; Barton, D. N.; Bates, M. E.; Linkov, I. Use ofstochastic multi-criteria decision analysis to support sustainablemanagement of contaminated sediments. Environ. Sci. Technol. 2012,46 (3), 1326−1334.(34) Knutzen, J.; Bjerkeng, B.; Næs, K.; Schlabach, M. Polychlori-nated dibenzofurans/dibenzo-p-dioxins (PCDF/PCDDs) and otherdioxin-like substances in marine organisms from the Grenland fjords,S. Norway, 1975−2001: Present contamination levels, trends andspecies specific accumulation of PCDF/PCDD congeners. Chemo-sphere 2003, 52 (4), 745−760.

(35) Barton, D. N.; Navrud, S.; Bjørkeslett, H.; Lilleby, I. Economicbenefits of large-scale remediation of contaminated marine sedi-mentsA literature review and an application to the Grenland fjordsin Norway. J. Soils Sediments 2010, 10 (2), 186−201.(36) Saloranta, T. M.; Armitage, J. M.; Haario, H.; Næs, K.; Cousins,I. T.; Barton, D. N. Modeling the effects and uncertainties ofcontaminated sediment remediation scenarios in a Norwegian Fjordby Markov chain Monte Carlo simulation. Environ. Sci. Technol. 2007,42 (1), 200−206.(37) Brans, J. P.; Vincke, P.; Mareschal, B. How to select and how torank projects: The PROMETHEE method. Eur. J. Operat. Res. 1986,24, 228−238.

Environmental Science & Technology Article

dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. XXXX, XXX, XXX−XXXH