sources of uncertainty in model predictions: lessons learned from the iaea forest and fruit working...

Journal of Environmental Radioactivity 84 (2005) 297e314

www.elsevier.com/locate/jenvrad

Sources of uncertainty in model predictions:lessons learned from the IAEA Forest

and Fruit Working Group modelintercomparisons*

Igor Linkov a,*, Dmitriy Burmistrov b

a Cambridge Environmental Inc., 58 Charles Street, Cambridge, MA 02141, USAb Menzie-Cura and Associates, Inc., Winchester, MA, USA

Received 14 November 2002; received in revised form 16 October 2003; accepted 17 October 2003

Available online 22 June 2005

Abstract

The International Atomic Energy Agency (IAEA), through the BIOMASS program, hasprovided a unique international forum for assessing the relative contribution of different

sources of uncertainty associated with environmental modeling. The methodology andguidance for dealing with parameter uncertainty have been fairly well developed andquantitative tools such as Monte-Carlo modeling are often recommended. The issue of model

uncertainty is still rarely addressed in practical applications and the use of several alternativemodels to derive a range of model outputs (similar to what was done in IAEA model inter-comparisons) is one of a few available techniques. This paper addresses the often overlooked

issue of what we call ‘modeler uncertainty,’ i.e., differences in problem formulation, modelimplementation and parameter selection originating from subjective interpretation of theproblem at hand. This study uses results from the Fruit and Forest Working Groups created

* This paper uses the results of the following models: SPADE (Mitchell, Ould-Dada), FRUTICROM

(Robles), FRUITPATH (Linkov, Burmistrov), RUVFRU (Eged), DOSDIM (Sweeck), ASTRAL

(Mourlon), FORESTLAND (Avila and Moberg), FOA (Bergman), FORM (Frissel), FORWASTE

(Konoplev and Bulgakov), FORESTPATH (Linkov and Burmistrov), ECORAD (Mamikhin), RIFE

(Shaw), FORESTLIFE (Dvornik and Zhuchenko), RODOS (Calmon), FINNWOOD (Rantavaara),

LOGNAT (Scimone).

* Corresponding author.

E-mail address: [email protected] (I. Linkov).

0265-931X/$ - see front matter � 2005 Elsevier Ltd. All rights reserved.

doi:10.1016/j.jenvrad.2003.10.009

mailto:[email protected]

http://www.elsevier.com/locate/jenvrad

298 I. Linkov, D. Burmistrov / J. Environ. Radioactivity 84 (2005) 297e314

under the BIOMASS program (BIOsphere Modeling and ASSessment). The greatestuncertainty was found to result from modelers’ interpretation of scenarios and approxima-

tions made by modelers. In scenarios that were unclear for modelers, the initial differences inmodel predictions were as high as seven orders of magnitude. Only after several meetings anddiscussions about specific assumptions did the differences in predictions by various models

merge. Our study shows that the parameter uncertainty (as evaluated by a probabilisticMonte-Carlo assessment) may have contributed over one order of magnitude to the overallmodeling uncertainty. The final model predictions ranged between one and three orders of

magnitude, depending on the specific scenario. This study illustrates the importance ofproblem formulation and implementation of an analyticedeliberative process in fate andtransport modeling and risk characterization.

� 2005 Elsevier Ltd. All rights reserved.

Keywords: Radionuclide; Modeling; Uncertainty; Model intercomparisons; Forests; Fruit; Bioaccumu-

lation

1. Introduction

The estimation of exposure point concentrations, used in assessing risks toecological receptors and humans, often requires modeling. Various models can beused to predict contaminant concentrations in a given ecosystem compartment usingdata collected in the field or available from the literature for other studiedcompartments. For example, contaminant concentrations in plants or fruits areoften calculated using measured data on soil contamination and simplistic transferfactors. Another common use of models is the prediction of temporal changes inexposure point concentrations, especially projections into future dates, given onlylimited sampling data collected in the field. Often models are applied to assess therisk reduction from different remedial and/or abatement scenarios developed as partof remedial investigations.

It is well recognized that model predictions can be highly uncertain, anduncertainty analysis has become an integral part of risk assessment, required bymany regulatory agencies. Taxonomies of uncertainty are presented in severalpublications (Cullen and Frey, 1999; US EPA, 2001; Morgan and Henrion, 1990).The following three broad categories are usually defined as:

� Parameter uncertainty e uncertainty in the value of an input parameter ina model.

� Model uncertainty e uncertainty about a model structure (i.e., the relevance ofsimplifying assumptions and mathematical equations).

� Scenario uncertainty e uncertainty regarding missing or incomplete informationto fully define the model.

Regulatory agencies and individual scientists have developed guidance andapproaches to the quantification of uncertainties in model predictions (Cullen, 2002;

299I. Linkov, D. Burmistrov / J. Environ. Radioactivity 84 (2005) 297e314

Evans et al., 1994; Linkov et al., 2002). In most cases, these approaches dealwith uncertainty resulting from model parameters, while the question of modeluncertainty is discussed less frequently (Cullen and Frey, 1999). An even less studiedarea is scenario uncertainty.

This study is an attempt to quantify and compare uncertainties in the modelingprocess, including what we call ‘modeler uncertainty’, originating from the subjectiveinterpretation of the problem at hand. ‘Modeler uncertainty’ could be classified as anadditional dimension within the scenario uncertainty since the lack of completespecification of the scenario results in the need for the modeler to make choices ashow to fill the missing details. In the case of this study, all efforts were made toclarify the scenario and modelers were encouraged to request for additional data.Thus ‘modeler uncertainty’ rather than scenario uncertainty is likely to be the driverin this study.

Assessment of climate change is one of a few areas where rigorous assessment ofboth model uncertainty and model and data interpretation by experts has beenconducted. Global coupled oceaneatmosphere circulation models (Covey et al.,2003) and three-dimensional ocean models (Orr et al., 2001) are just a few examplesof large physical model intercomparison studies that are currently underway. Policymodels related to climate change and associated uncertainties, as well as model anddata interpretation by climate change experts, are also being actively researched(Casman et al., 1999; Morgan et al., 2001; Morgan and Keith, 1995). These andother studies (Brown et al., 1997; Budnitz et al., 1995; Evans et al., 1994; Kann andWeyant, 2000) have illustrated the importance of uncertainty analysis and haveresulted in specific recommendations regarding the improvement of risk analysis(Fineberg and Stern, 1996; Morgan, 1998). The National Research Council (Finebergand Stern, 1996) views risk characterization as an analytical and deliberative processthat must be mutual and recursive and should include early and explicit attention toproblem formulation.

Even though the issues of model uncertainty and judgmental biases by expertshave been addressed in the climate change area, the quantification of ‘modeleruncertainty’ in smaller-scale human health and ecological risk assessments has notbeen fully addressed. The critical differences between climate change and en-vironmental assessments conducted for Superfund and other contaminated sites aresmaller modeling time frames (usually years vs. centuries and sometimes millennia inthe case of climate change), as well as often-severe budgetary constraints on the dataanalysis and modeling. Moreover, climatic models are all calibrated using essentiallythe same dataset (past historical data), while most of the fate and transport andexposure assessment models use different data for model testing and calibration. Ourgoal is to illustrate the overall uncertainty reduction achieved as a result ofimplementing a deliberative process with respect to modeler uncertainty, as well asthe limits of the predictive ability of environmental models. In other words,incorporation of stakeholder opinion and close consultation with decision makersand other parties should start early in the process of environmental assessment.

The IAEA, through the BIOMASS program, has provided a unique internationalforum for activities aimed at increasing confidence in methods and models used for


risk assessment related to environmental releases. As part of the program, workinggroups on several themes were set up: dose reconstruction for long-term tritiumdispersion in the environment; radionuclide uptake by fruits; and radionuclidemigration and accumulation in forest ecosystems. In each group, IAEA, through itstechnical secretariat, developed several scenarios that described specific cases ofcontaminant releases, and then tasked individual modelers or groups of modelerswith the prediction of future contaminant redistribution and, in some cases, risksassociated with contamination. The overall objective of each working group was tocompare the results produced for the same endpoint by different models. In modeledata intercomparisons, these model predictions were eventually compared with themeasured field data. In modelemodel intercomparisons, only the differences amongmodel predictions were evaluated.

The design of the program provided an opportunity to address model uncertaintyexpressed as the relative difference in predictions by models. The ranges ofpredictions reported by individual models can be used as measures for parameteruncertainty. Finally, the design of the BIOMASS program, which included severaliterations in modeling (each one after a team meeting where the scenario wasdiscussed), has allowed us to assess the degree to which the interpretation of thescenario by each individual modeler affects the model predictions. Even thoughseparate consideration of uncertainty and variability (i.e., heterogeneity in nature) isimportant in environmental policy assessment, because each has very differentimplications for decision making (Morgan and Henrion, 1990; Linkov et al., 2001),the IAEA exercise was not intended to study this issue and variability anduncertainty are not distinguished in this study.

2. Study design

This paper presents results for two modelemodel intercomparisons and onemodeledata intercomparison conducted by the Fruit Working Group and twomodelemodel intercomparisons conducted by the Forest Working Group. Details ofthe study design can be found in IAEA (2002, 2003) and in other papers presented inthis volume. The study was conducted for a set of idealized scenarios that describedthe characteristics and timing of an environmental release, its deposition pattern,local vegetation types and agricultural settings, and soil properties.

As discussed in IAEA (2003), two scenarios were developed for modelemodelintercomparison in the Fruit Working Group. One scenario assumed spike depo-sition (i.e., deposition occurring in a very short time, as in the case of accidentalreleases) while another scenario assumed continuous deposition (e.g., what mightoccur during a routine operation of nuclear facilities). One of the forest scenariosalso assumed spike deposition and was loosely based on the experience in Chernobylforests (IAEA, 2002). Another scenario assumed a continuous subterranean source(waste scenario). These scenarios were hypothetical, but based on data sets from realfruit and forest ecosystems. Calculation endpoints for modelemodel intercompa-risons were a time series of radionuclide concentrations in fruits (strawberries,apples, blackberries, blueberries, mushrooms), soil, and plants.


The Fruit Working Group modeledata intercomparison uses results from anexperimental study conducted specifically for the BIOMASS program (IAEA, 2003).The experimental work was carried out at Universita Cattolica del Sacro Cuore ofPiacenza (Italy). Strawberry plants were grown in pots filled with a peat substrateand were sprayed by solutions containing 134Cs and 85Sr in April. Strawberries wereharvested twice (in June and July). The calculation endpoints were assessed forradionuclide concentrations in fruits and soil at each harvest date.

3. Participating models

Participating models are described in detail in other papers of this volume and inIAEA publications (IAEA (2002, 2003)). Table 1 highlights the differences andsimilarities for the six models utilized in the Fruit Working Group. Five of thesemodels were developed by the government agencies for regulatory assessment ofradionuclide concentrations in fruits, as well as risk resulting from their con-sumption. In general, the models have different approaches to the most importantprocesses for radionuclide transport within plants. Some of the processes though aretreated similarly by most of the models. For example, the interception fraction isused by five out of six participating models, while SPADE uses the deposition rate.Radionuclide loss from vegetation was modeled using residence half-time (i.e., first-order differential equation) by four models. SPADE models the radionuclide lossesby considering external and internal plant surface layers and thus the loss dynamic ismore complex. RUVFRU models losses from vegetation by using weathering andresuspension factors.

Other redistribution processes in fruit ecosystems are handled very differently bythese models. SPADE considers translocation among seven vegetation compart-ments, while DOSDIM and ASTRAL consider translocation to fruits only.FRUITPATH and RUVFRU do not consider these processes, and model the fruitas a part of plant. Radionuclide migration in soils is another area of divergence inmodeling. FRUITPATH and RUVFRU consider labile and fixed pools ofradionuclides in soil, where sorption and desorption processes occur. Radionuclidehalf-times are used in their modeling of these compartments. DOSDIM and SPADEconsider several vertical depth strata. Radionuclide transfer between the strata isdescribed by empirical transfer factors. These models use distribution coefficients tomodel radionuclides in soil solutions.

The degree of treatment of temporal processes is one of the major factors varyingacross the models. For example, only two models (RUVFRU and SPADE) explicitlyconsider plant growth, while the others assume constant plant biomass. In-corporation of time-varying parameters also varies across the models. Four modelsare designed to provide point estimates for concentrations and risk. The DOSDIMmodel is capable of incorporating limited stochastic calculations, while FRUIT-PATH is the only wholly probabilistic model that incorporates probabilisticMonte-Carlo simulations and predicts a probability distribution for radionuclideconcentrations at any moment in time.

Table 1

Model representation of major processes

Radionuclide

speciation

Uptake

from soil

Partition in

soil. Uptake

by plant at

vegetation

surface

Root distribution

in profile used to

calculate depth

dependant root

uptake rate

constants

nt

rs

e

Not considered Equilibrium transfer

factor for root

uptake

r,

ep

Sorption and

desorption

within labile

and fixed soil

compartments

First-order process

Sorption and

desorption

within labile

and fixed soil

compartments

First-order process

with corrections

for moisture

content

302

I.Linkov,

D.Burm

istrov/J.Enviro

n.Radioactivity

84(2005)297e

314

Model Deposition Loss from

vegetation

Translocation

within

vegetation

Soil

transport

SPADE Ground

deposition

rate and air

concentration.

Deposition to

fruit not

considered

From vegetation

surface to internal

plant and soil (split

between solution

and organic)

Translocation

between roots,

stem, storage

organ, internal

leaf, external leaf

and fruit considered

Soil solution,

inorganic and

organic matter

considered in 10

layers of 3 cm

depth each.

Considers effect

of inorganic and

organic matter

distribution in

soil profile

FRUTICROM Interception

fraction for

fruit and leaves

Decrease rate due

to radioactive decay,

growth and pruning

Translocation from

external plant surface

on to internal part

plant considered

No transport inside

the soil compartme

considered. Conside

losses by radioactiv

decay, leaching and

other processes

FRUITPATH Interception

fraction (fruit

and leaves)

First-order process

with specified half-time

Not considered First-order process

among organic laye

labile, fixed and de

soil compartments

RUVFRU Interception

fraction (fruit

and leaves)

Weathering and

resuspension factors

Not considered First-order process

among four soil

compartments

DOSDIM Interception First-order

differential

equations for

translocation,

weathering

Only translocation

to fruit considered

First-order process

in root zone (loss

from root zone

through leaching)

Exchange between

soil solution and

solid soil phase

given by distribution

coefficient

Equilibrium transfer

factor for root

uptake

Decrease rate

due to

weathering

Translocation to

the edible organ

(in previously

mentioned

aggregated

transfer factor)

No transport inside

the soil compartment

considered. Ploughing

taken in consideration

and giving homogeneous

concentration

A decrease rate for

bio-availability in

soil (fixation and

migration in soil)

is radionuclide

dependant

Transfer factor from soil

concentration to edible

organ concentration

(Z aggregates root absorption

and translocation processes)

303

I.Linkov,

D.Burm

istrov/J.Enviro

n.Radioactivity

84(2005)297e

314

fraction (fruit,

not leaves)

ASTRAL Interception

fraction (as an

aggregated

transfer factor

to leaves)


Uptake of radionuclides from soil is the area of largest uncertainty and thus itcreates the most controversy among modelers. FRUITPATH and RUVFRU useplant uptake rates for a labile soil compartment. DOSDIM and ASTRAL use anequilibrium transfer factor. SPADE uses rate constants from 10 vertical soil strata tocreate a specified distribution for plant roots.

4. Results for the Fruit Working Group

Results of the modelemodel intercomparisons in the Fruit Working Groupincluded time series for radionuclide concentration in strawberries, blackberries andapples, and other compartments predicted over 10 years, following the beginning ofchronic or acute radionuclide deposition. In this paper, we present analyses ofmodeled cesium concentrations in strawberries for two discrete time periods: 1 and10 years following deposition (Figs. 1 and 2). Similarly, only results for fruits arereported for the modeledata scenario (Fig. 3). To facilitate comparison, concen-trations predicted by models were normalized to the median of predicted valuefor modelemodel intercomparisons, and were normalized to measured values formodeledata intercomparisons. Results are presented using a logarithmic scale. Inaddition to concentration in fruits, IAEA (2003) reports radionuclide concentrationsin plant (stem, leaves, wood) and soil compartments for all modeled radionuclides.

Fig. 1 presents results for the acute deposition scenario. In the first two rounds ofcalculations, the model predictions range over four orders of magnitude for short-term periods following the release (Fig. 1, top plot) and over four orders ofmagnitude for the long-term periods (Fig. 1, bottom plot). Even though one mightexpect that the short-term modeling predictions would be more variable due todifferent approaches to the non-equilibrium condition directly following deposition,the magnitude of the difference was striking. After two rounds of discussions, themodelers agreed that the major differences are resulted from differential treatment ofair dispersion by modelers. For example, some modelers assumed that the totalradioactivity released in the atmosphere reached the ground locally, while othersmodeled radionuclide resuspension over larger areas. Because in many practicalsituations deposition per unit area is measured and participating models weredesigned to deal with this parameter, the scenario was modified by specifying thesource term in units of radionuclide flux per unit of area for the third and fourthround of calculations, rather than providing a general atmospheric radionuclideconcentration in the atmosphere as the source term. As a result, models reachedmuch better consistency (two orders of magnitude), even though predictions by anadditional model (RUVFRU) were added. As expected, the short-term variationremained (slightly) higher than the long-term predictions.

Even though the modeling for the continuous deposition scenario (Fig. 2) wasconducted following the first two rounds of the acute deposition scenario whenmany scenario-related uncertainties were resolved (e.g., the source term wasspecified in the radionuclide flux units), the variation across models was quite high(three to four orders of magnitude) for the first round of calculations. During


the first meeting, the modelers agreed that the differences were resulted from thedifferent definitions of ‘continuous deposition’ used by modelers. Most of themodels were developed for acute deposition scenarios, and the notion of continuousdeposition was new for many modelers. After defining a common interpretation,models reached an excellent consistency in prediction; the final results are all withinone order of magnitude.

FRUITPATH was the only fully probabilistic model in the group, and it providednot only mean values, but also percentiles for a distribution of predicted radionuclideconcentrations resulting from input parameter uncertainty. The uncertainty distribu-tion for FRUITPATH model output results from the FRUITPATH parameteruncertainties and is much narrower than the difference in predictions among allparticipating models (i.e., model uncertainty).

0.000001

0.00001

0.0001

0.001

0.01

0.1

1

10

100

1 2 3 4

Model Run

Ratio

to

th

e M

ed

ian

C

alcu

latio

n

1 year after

deposition

RUVFRU

SCKCEN

IPSN

CHECOSYS

SPADE

FRUITPATH

CIEMAT

0.0001

0.001

0.01

0.1

1

10

100

1 2 3 4

Model Run

Ratio

to

th

e M

ed

ian

C

alcu

latio

n

10 years after

deposition

RUVFRU

SCKCEN

IPSN

CHECOSYS

SPADE

FRUITPATH

CIEMAT

Fig. 1. Results of modelemodel intercomparisons (Fruit WG): acute deposition scenario. Ratio of cesium

concentration in strawberries predicted by the participating models for the 1st (top plot) and 10th (bottom

plot) year following the acute deposition to the median predicted value is presented.


The experimental study used in the modeledata intercomparisons and theassociated scenario was designed in the following three working group meetings. Bythat time, participating modelers had developed understanding of scenarioassumptions and it was no surprise that model predictions were very close to eachother (Fig. 3). The first round of calculations was blind, i.e., experimental data werenot released. During this meeting, IAEA encouraged modelers to discuss the reasonsfor possible differences among models. Moreover, IAEA informed that the range ofmodel predictions does not capture the experimental data, but no data were releasedto the modelers. The modelers agreed that many models did not properly calculatethe radionuclide concentration in strawberries harvested just a few days followingradionuclide deposition, but the correction was not expected to result in significant

1 2 3 4

Model Run

0.1

1

10

100

Ratio

to

th

e M

ed

ian

Calcu

latio

n

1 year after

deposition

SCK-CEN

FRUITPATH -95%

FRUITPATH

FRUITPATH +95%

SPADE

CIEMAT

0.1

1

10

100

1000

1 2 3 4

Model Run

Ratio

to

th

e M

ed

ian

Calcu

latio

n

10 years after

deposition

SCK-CEN

FRUITPATH -95%

FRUITPATH

FRUITPATH +95%

SPADE

CIEMAT

Fig. 2. Results of modelemodel intercomparisons (Fruit WG): continuous deposition scenario. Ratio of

cesium concentration in strawberries predicted by participating models for the first (top plot) and 10th

(bottom plot) year following the beginning of continuous deposition to the median predicted value is

presented.


concentration differences. In the second round of calculations, better consistencywith experimental data was achieved. A slightly wider concentration spread in modelpredictions in the second round can be partially explained by the implementation ofmore sophisticated models, as well as by the participation of additional models.

As with the modelemodel intercomparisons, FRUITPATH was the only fullyprobabilistic model. The FRUITPATH model parameters were further adjusted forthe specific local conditions of the modeledata scenario and thus resulted in evensmaller parameter uncertainties in the model output. The model uncertainty again ismuch higher than the parameter uncertainty for the FRUITPATH model.

5. Results for Forest Working Group

Results of the modelemodel intercomparisons in the Forest Working Groupincluded time series for radionuclide concentration in trees, understory, berries,mushrooms, and soil compartments predicted over 20 years, following the beginningof acute radionuclide deposition and 200 years following the capping of disposed

0.01

0.1

1

10

0 1 2 3Model Run

Ratio

to

M

easu

rem

en

ts Measured (Bq/g)

SPADE

ASTRAL

CIEMAT

FRUITPATH-95%

FRUITPATH

FRUITPATH+95%

DOSDIM

RUVFRU

Srawberries,

First Harvest

0.01

0.1

1

10

0 1 2 3Model Run

Ratio

to

M

easu

rem

en

ts Measured (Bq/g)

SPADE

ASTRAL

CIEMAT

FRUITPATH-95%

FRUITPATH

FRUITPATH+95%

DOSDIM

RUVFRU

Second Harvest

Fig. 3. Results of modeledata intercomparison scenario (Fruit WG). Ratio of radionuclide concentration

in strawberries harvested in May (first harvest) and June (second harvest) to the modeled values. Foliar

deposition of Cs-134 occurred in April.


radioactive waste. In this paper, we present analyses of modeled cesiumconcentrations in wood 10 years following deposition for ‘Chernobyl’ scenarioand 10 years following the capping and forest planting event for the ‘‘Waste’’scenario (Fig. 4). Since only one or two model iterations were attempted by theForest Working Group, only the final results are presented in this paper. To facilitatecomparison, the results are presented using a logarithmic scale and eachparticipating model result is presented using the same symbol. In addition to theconcentration in trees, IAEA (2002) reports radionuclide concentrations in differenttree compartments, forest plants and soil compartments for all modeled radio-nuclides and scenarios.

Fig. 4 shows results for both scenarios. Even in the first rounds of calculations, themodel predictions are within one order of magnitude, which is quite different froma large range of variations observed in the Fruit Working Group. One of the reasonsis that many models were developed and calibrated using Chernobyl data and thescenario presented by IAEA was very familiar to the modelers. In contrast, the‘‘Waste’’ scenario that was not familiar for many modelers resulted in controversialinterpretation (as evident in the follow-up discussions) and results range over severalorders of magnitude, similar to the situation in the Fruit Working Group.

6. Discussion

6.1. Differences in model structures

The real-world complexities of the environment can be approximated differentlyby modelers and thus result in varying modeling assumptions and structures. Asimple system of agricultural fruits, and fate and transport of contaminants within

0.1

1

10

100

1000

Rad

io

nu

clid

e C

on

cn

tratio

n (B

q/kg

)

Waste Scenario

"Chernobyl"

Scenario

Trees

10 years after

deposition

Fig. 4. Results of modelemodel intercomparisons (Forest WG): acute deposition and chronic

(subterranean) source scenarios. Each dot corresponds to radionuclide concentration in trees 10 years

after contamination as predicted by one of the participating models.


this system, was approached very differently by the six models participating in theFruit Working Group. Similarly, very different approaches were taken by the forestmodels. Some of the models have dozens of compartments, representing multiple soilhorizons and individual components of plants, while others aggregate compartmentsbroadly. The number of individual input parameters required by the models varyfrom hundreds to just a dozen. Some models are designed to be deterministic andthus use only point estimates for the model parameters, while the probabilisticmodels incorporate parameters as probability distributions. The temporal dynamicsof radionuclide migration are treated as an equilibrium process by some models, andas a dynamic process modeled using differential equations by others.

The reasons leading to differences in model assumptions and model structuresimplemented by each research group came from: (i) model objectives; (ii) com-putational capacities; (iii) data available for model calibration and testing; and (iv)the scientific knowledge and technical and computational expertise available to eachresearch group at the time of model development. The majority of the models thatparticipated in the study were developed by government agencies and thus may havebeen designed to be conservative (though many assumptions and parameters werechanged to address specific scenarios). Several models have a long history ofdevelopment, starting in the 1980s, when computational capabilities were limited.Additionally, even though at that time the scientific understanding of the basicprocesses of radionuclide accumulation in plants was in general similar to what weunderstand now, many details regarding radionuclide speciation and migrationbecame better described and understood only in the mid-1990s, after the Chernobylaccident. Finally, many models have evolved from a description of a specific problemfaced by a research group in addition to a limited set of experimental data collectedin-house, while others were designed to be generic, i.e., have the ability to in-corporate multiple sets of experimental data for different ecosystems and species.

Even though the approaches taken by the research groups were very different, therange of model predictions for the final runs of modelemodel and modeledataintercomparisons was within one to two orders of magnitude. Given the high naturalvariability in radionuclide distribution and the fact that the models were notspecifically calibrated for the hypothetical scenario, this range can be characterizedas quite narrow. It is probably indicative of the ‘‘true’’ model uncertainty that resultsfrom different model forms and simplifications made by each research group. Ifmodels are properly adjusted to the site conditions and the general assumptionsabout a specific scenario are similar, model uncertainty can be much less than theuncertainty resulting from subjective interpretation (or misinterpretation) of theproblem or scenario at hand by individual modelers and/or research groups.

6.2. Differences in scenario interpretation

Results of this study show that initial interpretation of the scenario by modelersmay have resulted in model outcomes ranging over six orders of magnitude in thiscase. Even though the very first modelemodel intercomparison scenario was puttogether by a group of several technical experts with support from IAEA and was


peer-reviewed prior to the release, the interpretations made by the research groupswere quite different. It took two two-day face-to-face meetings to develop commoninterpretations of the scenario. This deliberation process not only resulted in asignificant reduction of spread of model prediction in the first acute release scenario,but also developed a common language that allowed the technical secretariat of theworking group and IAEA to develop two other scenarios that were more consistentand understandable for the modelers. The spread of the modeling results for thesecond modelemodel intercomparison scenario (continuous source term) was lessthan three orders of magnitude, while just a little over one order of magnitude spreadwas reported in the last modeledata intercomparison exercise.

6.3. Parameter uncertainty

In the Fruit Working Group, the parameter uncertainty analysis was conductedfor the FRUITPATH model. Monte-Carlo simulations were used. The uncertainmodel parameters were represented as probability distributions based on a literaturereview. The available literature data were limited to the ecosystems close to the siteunder consideration in a specific scenario. Results of a literature review show thatvalues for model parameters are very uncertain; for many parameters, just one ortwo measurements were reported. The majority of the parameters were thusrepresented by triangular shape distributions characterized by three parameters:minimal and maximal values, and mode.

The parameter uncertainty for the FRUITPATH model, measured as the 95thpercentiles for a distribution of predicted radionuclide concentrations (uncertaintyand variability combined), was found to vary within one order of magnitude. This ismuch less than the model uncertainty and ‘modeler uncertainty’ presented above.Such a narrow range is often observed in models when continuous distributionsare used, and can be explained by the heuristic procedures used to select modelparameters under uncertainty (Morgan and Henrion, 1990). Some of the heuristicsand their effects on this study are discussed below.

6.4. Reasons for model and modelers disagreement and consensus

Our study shows that models and modelers disagreed significantly in their initialcalculation attempts, but reached consensus after the analyticedeliberative processimplemented by the IAEA Working Group. The reasons for the disagreement andthen consensus can be explained by cognitive tendencies widely recognized in thefield (Kahneman et al., 1982; Morgan and Henrion, 1990). To make judgement inthe presence of uncertainty, people use approximate heuristic procedures of avail-ability, representativeness, anchoring and adjustment that sometimes lead to biasedoutcomes.

Many modelers and research groups that participated in the project had carriedout modeling for specific ecosystems and/or release situations prior to theirinvolvement in the IAEA project. Their interpretation of the first scenario andselection of the model parameters were driven to a large extent by their previous


experiences, i.e., they probably resorted to the heuristic procedure of availability. Thecontinuous deposition scenario for modelemodel intercomparisons was judged bymany modelers in reference to the acute deposition scenario that was characteristicof the Chernobyl accident, as well as of many of the model applications attempted byparticipants in the past. The representativeness of the acute deposition scenario andthe available data for the case of continuous deposition may have resulted in a largedifference among models during the first round of modelemodel intercomparisonsfor the continuous deposition scenario. Since very little information was availablefor model parameters, anchoring and adjustment may have biased estimates of modelparameters and distributions used in the study. This is likely to be the case for thetriangular distributions developed for the probabilistic model FRUITPATH: manyparameters had only a few available measurements, and the range of variationswas often influenced by an initial reference value. This can explain the narrowdistributions for FRUITPATH output, i.e., overconfidence by the FRUITPATHmodelers.

7. Conclusions

The IAEA model intercomparison study has provided a unique opportunity totest models, using a relatively simple and controlled ecosystem of agricultural fruitsand more complicated system of forests. The results of the study show that even forthese ecosystems and well-controlled deposition scenarios, the differences in modelpredictions may be quite high. In this exercise, the differences among models were ashigh as seven orders of magnitude for short-term predictions following the acuteradionuclide deposition.

The National Research Council (Fineberg and Stern, 1996) defines riskcharacterization as the outcome of an analyticedeliberative process, which dependscritically on systematic analysis and treatment of the problem. Our study confirmsthe crucial need for such a process in addressing risk characterization and modeling.The analyticedeliberative process implemented in this study significantly reduced theexpert biases and developed a coherent scenario interpretation. Since the modelerswere allowed to request all necessary information to clarify the specific scenarios,we believe that the different interpretation of the problem and biased problemformulation were the largest sources of uncertainty quantified in this study. Ourstudy shows that just one or two meetings of several technical experts may besufficient to resolve the differences and reach consensus.

A complicating factor in this interpretation is that the scenarios can never besufficiently specified and subjective interpretation will always take place. A goodcontrolled experiment that could test the magnitude of scenario interpretation maybe to give the same analytical model and scenario to several groups of modelers andto compare the results.

The differences among models (i.e., model uncertainty) seem to be much higherthan parameter uncertainties for a given model. FRUITPATH was the only modelthat provided not only point estimates for median or mean concentrations, but also


corresponding concentration ranges for parameter uncertainties. The parameteruncertainty predicted by FRUITPATH is within one order of magnitude, muchnarrower than model uncertainty, which can be characterized by a spread of severalorders of magnitude.

The large difference among modelemodel predictions reveals limitations in ourcurrent abilities to predict future concentrations of contaminants in fruits insituations where no prior site-specific measurements are available. Though rarelyimplemented, policy decisions for these conditions should rely on the results ofseveral modeling approaches. If only one model is used and only limited site-specificdata are available, the results of model predictions may differ from each other byseveral orders of magnitude.

Even though model predictions differ greatly for the generic modelemodelintercomparison scenarios, modeledata intercomparisons of short-term predictionsfor radionuclide accumulation by strawberries were satisfactorily modeled, withuncertainty within one order of magnitude. One of the reasons is that the modeledata scenarios presented data for calibration of deposition parameters as well asextensive information for site-specific adjustment of participating models. Moreover,most of the models were calibrated using similar experimental data.

It should be noted that the consistency in model predictions achieved after severalmeetings may not necessarily be reflective of the reduction in model uncertainty. Agroup of modelers could all reach a consensus on how to model a particularphenomenon, and thereby all of the modelers could end up with predictions that arevery similar to each other. If the consensus was in fact wrong (e.g., all the modelsincorrectly approach the phenomenon) then no uncertainty reduction is achieved. Inthis case, the uncertainty of the models (i.e., difference in absolute value of modelpredictions) may be small but the prediction as a whole would be biased.

Other complicating issues are the compensating errors in the calibration. In thisstudy consensus reached was not about the model structure, but rather about thescenario and all models stayed very different through the end of the study. Thecompensation errors were likely to be low since the models were calibrated in a sensethat they were better tailored to a specific study/scenario rather than mathematicallyfitted to specific experimental results. Very little difference among model predictionsin the ‘Chernobyl’ scenario may be attributed to similar modeling approaches andsomewhat overlapping calibration data sets. The model uncertainty could still behigh as in the unfamiliar ‘Waste’ scenario for forests.

The results of the modeledata intercomparison scenario show that selection ofthe best way to achieve maximum uncertainty reduction is a challenging task. Theresults of the modeledata intercomparisons show that if site-specific data (such asconcentration measurements) are available for model calibration, every modelperforms reasonably well and provides predictions within one order of magnitudefrom actual measurements. Probabilistic models calibrated using Bayesian techni-ques (Bates et al., 2003; Linkov et al., 1999) can perform especially well. In anotherstudy (Linkov and Burmistrov, 1999) we show that if measurements for the firstharvest are used for FRUITPATH model calibration, the model predictions for thesecond harvest can be very close (within factor of two) to actual field measurements.


It is important to notice that even if measurements for one compartment are used forcalibration, the results for all compartments can be improved. A similar observationwas recently made by Cullen (2002) in her study of the use of compartmental modelswith varying degrees of complexity. We have therefore recommended that IAEAconsiders modelemodel and modeledata intercomparison scenarios that providea limited set of data for model calibration in the next round of intercomparisonsunder future IAEA programs.

Acknowledgment

The authors are grateful to Granger Morgan, Mitch Small, Owen Hoffman, AmyRosenstein and one anonymous reviewer for their review and useful suggestions.Special thanks to Ansie Venter for her technical assistance with paper preparation.We appreciate careful review and assistance by Arun Varghese. The financialsupport of travel expenses for one of the co-authors by IAEA is gratefullyacknowledged. The work was done as part of the IAEA BIOMASS Program.

References

Bates, S.C., Cullen, A.C., Raftery, A.E., 2003. Bayesian uncertainty assessment in multicompartment

deterministic simulation models for environmental risk assessment. Environmetrics 44, 355e371.

Brown, J., et al., 1997. Probabilistic Accident Consequence Uncertainty Analysis: Food Chain

Uncertainty Assessment. Office of Publications of the European Communities, BrusselseLuxembourg,

Brussels.

Budnitz, R.J., Apostolakis, G., Boore, D.M., Cluff, L.S., Coppersmith, K.J., Cornell, C.A., Morris, P.A.,

1995. Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and the

Use of Experts. Lawrence Livermore National Laboratory. URCL-ID-122160, Livermore, CA.

Casman, E.A., Morgan, M.G., Dowlatabadi, H., 1999. Mixed levels of uncertainty in complex policy

models. Risk Analysis 19, 33e42.Covey, C., Krishna, AchutaRao, K.M., Cubasch, U., Jones, P., Lambert, S.J., Mann, M.E., Phillips, T.J.,

Taylor, K.E., 2003. An overview of the results from the coupled model intercomparison project.

Global and Planetary Change 37, 103e133.

Cullen, A.C., Frey, H.C., 1999. Probabilistic Techniques in Exposure Assessment: A Handbook for

Dealing with Variability and Uncertainty in Models and Inputs. Plenum Press, New York.

Cullen, A., 2002. Comparison of model predictions and measurements for PCB concentration in the

vicinity of Greater New Bedford superfund site. Environmental Science and Technology 36, 2033e

2038.

Evans, J.S., Gray, G.M., Sielken Jr., R.L., Smith, A.E., Valdez-Flores, C., Graham, J.D., 1994. Using of

probabilistic expert judgment in uncertainty analysis of carcinogenic potency. Regulatory Toxicology

and Pharmacology 20, 15e36.

Fineberg, H.V., Stern, P.C., 1996. Understanding Risk: Informing Decisions in a Democratic Society.

National Academy Press, Washington, D.C.

International Atomic Energy Agency, 2002. Modelling the Migration and Accumulation of Radionuclides

in Forest Ecosystems. IAEA-BIOMASS-1, Vienna.

International Atomic Energy Agency, 2003. Modelling the Transfer of Radionuclides to Fruit. IAEA-

BIOMASS-5, Vienna.

Kahneman, D., Slovic, P., Tversky, A. (Eds.), 1982. Judgment Under Uncertainty: Heuristics and Biases.

Cambridge University Press, New York.


Kann, Weyant, 2000. Approaches for performing uncertainty analysis in large-scale energy/economic

policy models. Environmental Modeling and Assessment 5 (1), 29e46.

Linkov, I., Burmistrov, D., 1999. Bayesian updating techniques for reducing parameter uncertainty for

site-specific applications. SETAC Annual Meeting, Philadelphia.

Linkov, I., Burmistrov, D., Kandlikar, M., Schell, W.R., 1999. Reducing uncertainty in the radionuclide

transport modeling for the Chernobyl forests using Bayesian updating. In: Linkov, I., Schell, W.R.

(Eds.), Contaminated Forests: Recent Developments in Risk Identification and Future Perspectives.

Kluwer, Amsterdam.

Linkov, I., von Stackelberg, K., Burmistrov, D., Bridges, T.S., 2001. Ecological risk assessment:

uncertainty and variability from trophic transfer in management of contaminated dredged sediments.

Science of the Total Environment 274, 255e269.Linkov, I., Burmistrov, D., Cura, J., Bridges, T.S., 2002. Risk-based management of contaminated

sediments: consideration of spatial and temporal patterns in exposure modeling. Environmental

Science and Technology 36, 238e246.Morgan, M., Henrion, M., 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk

and Policy Analysis. Cambridge University Press, New York.

Morgan, M.G., 1998. Commentary: uncertainty analysis in risk assessment. Human and Ecological Risk

Assessment 4, 25e39.Morgan, M.G., Keith, D.W., 1995. Subjective judgments by climate experts. Environmental Science and

Technology 29 (10), 468e476.

Morgan, M.G., Pitelka, L.F., Shevliakova, E., 2001. Elicitation of expert judgments of climate change

impacts on forest ecosystems. Climatic Change 49, 279e307.Orr, J.C., Maier-Reimer, E., Mikolajewicz, U., Monfray, P., Sarmiento, J.L., Toggweiler, J.R., Taylor,

N.K., Palmer, J., Gruber, N., Sabine, C.L., Le Quere, C., Key, R.M., Boutin, J., 2001. Estimates of

anthropogenic carbon uptake from four 3-D global ocean models. Global Biogeochemical Cycles

15 (1), 43e60.

United States Environmental Protection Agency (USEPA), 2001. Risk assessment guidance for superfund,

vol. 3. Part A. Process for Conducting Probabilistic Risk Assessment.

sources of uncertainty in model predictions: lessons learned from the iaea forest and fruit working...

Documents