towards quantification of uncertainty in predicting water quality failures in integrated catchment...
TRANSCRIPT
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 4
Avai lab le a t www.sc iencedi rec t .com
journa l homepage : www.e lsev ie r . com/ loca te /wat res
Towards quantification of uncertainty in predicting waterquality failures in integrated catchment model studies
A.N.A. Schellart a,*, S.J. Tait b, R.M. Ashley a
a Pennine Water Group, Department of Civil & Structural Engineering, University of Sheffield, Mappin Street, Sheffield S1 3JD, UKb Pennine Water Group, School of Engineering Design and Technology, University of Bradford, Bradford, BD7 1DP, UK
a r t i c l e i n f o
Article history:
Received 13 August 2009
Received in revised form
29 April 2010
Accepted 4 May 2010
Available online 11 May 2010
Keywords:
Uncertainty
Integrated modelling study
Sewer emissions
Receiving water impact
Flow quality modelling
Water quality failure
* Corresponding author.E-mail addresses: a.schellart@sheffield.
Ashley).0043-1354/$ e see front matter ª 2010 Elsevdoi:10.1016/j.watres.2010.05.001
a b s t r a c t
This paper describes the development and application of a method for estimating uncer-
tainty in the prediction of sewer flow quantity and quality and how this may impact on the
prediction of water quality failures in integrated catchment modelling (ICM) studies. The
method is generic and readily adaptable for use with different flow quality prediction
models that are used in ICM studies. Use is made of the elicitation concept, whereby expert
knowledge combined with a limited amount of data are translated into probability distri-
butions describing the level of uncertainty of various input and model variables. This type
of approach can be used even if little or no site specific data is available. Integrated
catchment modelling studies often use complex deterministic models. To apply the results
of elicitation in a case study, a computational reduction method has been developed in
order to determine levels of uncertainty in model outputs with a reasonably practical level
of computational effort. This approach was applied to determine the level of uncertainty in
the number of water quality failures predicted by an ICM study, due to uncertainty asso-
ciated with input and model parameters of the urban drainage model component of the
ICM. For a small case study catchment in the UK, it was shown that the predicted number
of water quality failures in the receiving water could vary by around 45% of the number
predicted without consideration of model uncertainty for dissolved oxygen and around
32% for unionised ammonia. It was concluded that the potential overall levels of uncer-
tainty in the ICM outputs could be significant. Any solutions designed using modelling
approaches that do not consider uncertainty associated with model input and model
parameters may be significantly over-dimensioned or under-dimensioned. With changing
external inputs, such as rainfall and river flows due to climate change, better accounting
for uncertainty is required.
ª 2010 Elsevier Ltd. All rights reserved.
1. Introduction describes the advantages in using an integrated approach to
1.1. Integrated urban catchment modelling
Discharges from urban drainage catchments can have a major
impact on the quality of receiving surface waters. FWR (1998)
ac.uk (A.N.A. Schellart),
ier Ltd. All rights reserved
managing urban wet weather discharges whereby the sewer
system, the treatment plant and the receiving water are
considered as a single interconnected system. An Integrated
Catchment Model (‘ICM’) approach is therefore seen as an
important technique for managing the impact of drainage and
[email protected] (S.J. Tait), [email protected] (R.M.
.
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 43894
waste water systems on the environment (FWR, 1998; Rauch
et al., 1998; Muschalla et al., 2008; Willems, 2008). An ICM
study typicallyusesa rainfall generationmodel,a rainfall runoff
model, an urban drainage flow quantity and quality model,
a waste water treatment model and a receiving water impact
model. The sub-models used within an ICM study can range in
complexity from conceptual models (Willems, 2010), to
complex deterministicmodels that are composed of numerous
interlinked empirically calibrated equations describing
processes that affect water quality (Priestley and Barker, 2006;
Benedetti et al., 2005). Commercial software packages with
many linked deterministic equations are commonly used in
engineering practice (Priestley and Barker, 2006; Osborne and
Lau, 2003). These software packages are typically used
without consideration of the uncertainty involved in the solu-
tion of their numerous deterministically based equations.
1.2. Uncertainty in integrated urban catchment modelsand urban drainage models
Even though the need to deal more explicitly with uncertainty
of urban drainage systems is argued by Harremoes and
Madsen (1999), Bertrand-Krajewski (2006) and Ashley et al.
(2005), relatively few studies deal with the quantification of
uncertainty in urban drainage modelling (Willems and
Berlamont, 1999; Clemens and Von der Heide, 1999;
Thorndahl et al., 2008) and uncertainty in water quality
processes in urban drainage (Bertrand-Krajewski and Bardin,
2002; Kanso et al., 2003; Mourad et al., 2005; McCarthy et al.,
2008). Fewer studies deal with Integrated Catchment Models
that include water quality processes, Freni et al. (2008),
Mannina et al. (2006) and Willems (2008).
Mannina et al. (2006), Freni et al. (2008) and Thorndahl et al.
(2008) have all used the Generalised Likelihood Uncertainty
Estimation (GLUE) method developed by Beven and Binley
(1992) to evaluate overall uncertainty. Willems (2008) used
variance decomposition to split total prediction uncertainty
into contributions of various uncertainty sources and the
different conceptual models within an integrated catchment
model. The method described in this paper is different to the
methods mentioned above, and is known as a ‘forward
uncertainty propagation method’ in combination with
a model reduction method, instead of a ‘conditioning of
uncertainty on data’ method such as GLUE, as defined by the
decision tree for selecting an uncertainty methodology as
described by Pappenberger et al. (2006).
For complex hydrodynamic models of typical urban
drainage systems, computational resources may quickly
become a limiting factor when estimating uncertainty in the
model output, as described by Thorndahl et al. (2008).
Thorndahl et al. (2008) needed to use 10 personal computers
run for several weeks when applying the GLUE methodology
on a small catchment in Denmark. Conceptual models are
therefore often used, because of their relatively short
computational run times. Haydon and Deletic (2009) describe
that, even when using a simplistic modelling approach,
running Monte Carlo simulations can take a week of run time
per input variable or model parameter for a practical system.
In conceptual models it is also difficult to gather expert
judgement on parameter value ranges, as the parameters
used can have a “weak” physical meaning. Various
approaches have been developed to overcome the problem of
long model run times when estimating uncertainty in model
outputs. Benedetti et al. (2005) and Rousseau et al. (2001)
describe a method whereby a ‘probabilistic’ shell is built
around a deterministic model to quantify the uncertainty of
the model predictions. Khuri and Cornell (1987) describe the
principle of response surface methods, which can be used to
describe the solution of more complicated models. The
response surfacemethodologywas formally developed by Box
andWilson in the 1950’s (Box, 1954). A response database was
used by Dahal et al. (2005) to estimate the reliability of river
dikes on a tidal river. It has also been used by Schellart et al.
(2008, 2010) as a tool to calculate the uncertainty in sewer
sediment deposit depth predictions.
1.3. Classification of uncertainty
Researchers have described different classification systems to
identify uncertainty types (e.g. Harremoes and Madsen, 1999;
Slijkhuis et al., 1999; Korving, 2004). Many input parameters
that are used to describe natural quantities are not fixed
values. In this paper this type of uncertainty will be referred to
as model input uncertainty. Most water quality relationships
have been empirically calibrated using laboratory or field data.
These equations are then implemented in many model
studies, often without reference to the original circumstances
in which the equations were developed. There is always
a level of uncertainty as to how well these calibrated equa-
tions represented the original calibration data, and also how
large the uncertainty is in the original measured data. In this
paper this type of uncertainty will be referred to as model
parameter uncertainty. Finally, there are also uncertainty types
classified as ‘ignorance’ by Wynne (1992), which is non-
reducible and cannot be quantified. This subdivision of
uncertainty into these categories is pragmatic providing
a logical structure to organise the uncertainty analysis pre-
sented here. This paper concentrates on the model input and
model parameter related uncertainties, as these are quantifi-
able when sufficient data or expert knowledge is available.
1.4. Elicitation of expert knowledge
There is often limited data and model development is not
always well documented, however, relevant expert knowl-
edge may be available. If this knowledge is used, uncertainty
levels in model inputs and parameters can be estimated
through elicitation. Garthwaite et al. (2005) describe the
concept of eliciting probability distributions, or ‘the process of
formulating a person’s knowledge and beliefs of uncertain
quantities into a (joint) probability distribution for those
quantities’. Garthwaite et al. (2005) and Kadane and Wolfson
(1998) describe several methods for eliciting probability
distributions. Kadane and Wolfson (1998) also describe elici-
tation examples from a wide range of areas such as
economics, clinical trials, demography andmacro-economics.
O’Hagan (1998) describes two other elicitation examples,
future capital maintenance of water treatment works and
hydraulic conductivity of rocks at a potential nuclear waste
disposal site.
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 4 3895
1.5. Aim of this study
This paper describes the development of a method that esti-
mates uncertainty in complex deterministic ICM predictions
using prior elicited probability distributions combined with
a model reduction method such as a response database. This
method is applied to a small urban drainage catchment in the
UK to estimate the uncertainty associated with the predicted
number of water quality failures from combined sewer over-
flow discharges. Prior to the study described in this paper, an
ICM study using deterministic models had been used by the
sewer operator in order to assess compliance with receiving
water quality standards. This paper describes a ‘didactical’
example, where uncertainty in the sewer flow quality and
quantity component of the ICM is studied and described in
statistical terms using a form of elicitation. The uncertainty in
the sewer flow quality and quantity model inputs and model
parameters is used to estimate the range and number of water
quality failures. Monte Carlo simulations have been carried
out using a response database. The uncertainty analysis
methods used in this paper are not new in themselves, but
their application to estimate uncertainty in the outputs of an
ICM is novel. Although this paper only takes uncertainty in the
sewer flow quality and quantity model inputs and parameters
into account, the method is generic and can also be used to
assess uncertainty in the ICM output based on uncertainty in
other ICM components.
2. Description of case study and originaldeterministic ICM
An ICM study had been carried out by a UK sewer operator in
order to demonstrate compliance with the Fundamental
Intermittent Standards (FIS), FWR (1998), that are used to set
allowed discharges from Combines Sewer Overflows (CSOs) in
England. The calibration of the original ICMwas carried out to
current industrial standards, e.g. WaPUG (2002). The ICM for
this catchment comprised several sets of deterministic
equations embedded in different commercial software pack-
ages (Fig. 1). Data was transferred manually between these
software packages.
A 10-year future rainfall series, from 2010 to 2019 with a 5-
min resolution, had been derived for this catchment using the
stochastic rainfall generator STORMPAC (WRc, 2009). A future
Fig. 1 e Overview of the Integrated Catchment Mo
rainfall series had been used, as the ICM study was used to
determine capital investment requirements in order to restore
or maintain compliance with the FIS. The MIKE NAM package
(Rainfall runoff model; DHI, 2009) had been used to calculate
the river flow quantity just upstreamof the urban area and the
river reach being examined for compliance. The river catch-
ment rainfall runoff model, MIKE NAM (DHI, 2009) had been
calibrated, using the following flow calibration parameters:
Umax (maximum water content in surface storage); Lmax
(Maximum water content in root zone); CQOF (overland flow
runoff coefficient), CK12 (Time constant for routing interflow
and overland flow); TOF (Root zone threshold for overland
flow). These parameters were adjusted until the predictions of
the NAM model visually matched the observed river flow rate
data series and predicted and accumulated flows matched. A
comparison of 6-years of 15-min gauged river flow data with
modelled flow, showed that the modelled flow peaks fell
within a þ or �20% range of the measured maximum flows.
Infoworks CS (v 7.0) had been used to calculate runoff from
the urban catchment surfaces, flows in the sewer system and
hence the quantity and quality of combined sewer overflow
emissions and quantity of the WwTW effluent entering the
river. The combined sewer system serves an area of 294 ha
and has around 11,000 contributing inhabitants. The system
contained one Waste water Treatment Works (WwTW) and
five combined sewer overflows (CSOs), all discharging into the
same river. The Infoworks CSmodel was calibrated using flow
quantity and water depth data obtained from a short term
sewer flow survey. Three dryweather days and 6 storm events
were used, following the selection criteria recommended in
WaPUG (2002). The movement of the following pollutants had
been modelled: sediments; Biochemical Oxygen Demand
(BOD) in the dissolved phase as well as attached to sediment;
and unionised Ammonia (NH4þ). The case study Infoworks CS
model consisted of 505 conduits, 530 nodes and 5 CSO
structures.
A river impact assessment tool developed by Priestley and
Barker (2006) had been used for the assessment of the impact
on the receiving water downstream of the urban area. The
impacts of combined sewer overflow events and WwTW
effluent on the receiving water was estimated over a 4.5 km
river stretch downstream of the WwTW location. The river
impact model calculates river water depths and velocities,
assuming steady, uniform and fully turbulent flow. A Streeter-
Phelps typemodel (FWR, 1998) is used to estimate the decay of
delling procedure used in the test catchment.
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 43896
Ammonia, BOD and re-aeration within the river reach. The
input to the river impact model is (see also Fig. 1):
� A time series of river flow quantity at the upstream end of
the river reach for which compliance needs to be tested
(derived from the NAM model output).
� Mean and standard deviation of the concentration of flow
quality parameters at the upstream boundary derived from
30 samples taken in the summer periods over 5 years (BOD
mean 1.9 mg/L and standard deviation 1.75 mg/L, Ammonia
mean 0.001 mg/L and standard deviation 0.002 mg/L,
Ammonium 0.081 mg/L standard deviation 0.094 mg/L,
Dissolved Oxygen mean 8.86 mg/L standard deviation
1.355 mg/L, pH mean 7.76 and standard deviation 0.249,
Temperature 13.696 �C and standard deviation 1.868 �C).� A time series of WwTW effluent flow quantity and quality.
WwTW effluent flow quantity is derived from the Infoworks
CS model. WwTW effluent quality is derived from samples
(BOD mean 9.20 mg/L and standard deviation 3.82 mg/L and
NH4þ mean 1.26 mg/L and standard deviation 2.29 mg/L,
based on 181 samples taken between January 2000 and
August 2006).
� Flow quantity and quality of all sewer system spill events
(derived from Infoworks CS model).
� Geometric data and re-aeration parameters of the river
reach where compliance is to be tested (downstream of the
urban area). (Channel slope was 4.4 m/km, the channel
width 2.5 m, side slope 0.675 m/m and the Manning’s
roughness 0.04).
Based on the inputs described above, the river impact
model calculates the numbers of failures of the FIS for
a defined time period.
3. Uncertainty estimation method
The method used in this study for estimating levels of
uncertainty in the outputs of an ICM such as described in
Section 2 can be summarised as follows:
3.1. Mapping the processes within the integratedcatchment model
In order to create an overview of the deterministic modelling
relationships and their integration, all equations, inputs and
parameters are mapped. All possible sub-model outputs are
listed, as well as the outputs necessary for testing compliance
with water quality standards. As described by Rauch et al.
(1998), existing models used as sub-models for ICM studies
are often too complex, and not all their possible outputs need
to be available as inputs to other sub-models. It is therefore
important to clearly identify all the model input and model
parameter variables that influence the values of the outputs
required by an ICM study.
3.2. Initial sensitivity check
An initial sensitivity check is carried out to eliminate, from
further study, processes that will not contribute significantly
to the level of uncertainty in the outputs used for compliance
testing.
3.3. Elicitation of probability distributions of model andinput variables
Knowledge on the listed model inputs and model parameters
is collected from available experts, literature sources, and
available data. Garthwaite et al. (2005), Kadane and Wolfson
(1998) and O’Hagan (1998) describe a number of different
elicitation methods to translate this kind of knowledge into
probability distributions to describe uncertainty. Kadane and
Wolfson (1998) defined two different types of elicitation e
predictive and structural. In a predictive elicitation study
experts are asked about their opinion on the uncertainty of
a dependent variable, given an expected range of the predictor
variables. In a structural elicitation study, experts are asked
directly to define a priori distributions of the model and input
parameters. The method followed in this paper is structurally
based, experts were asked to define a confidence interval for
identified model inputs and model parameters. The details of
the elicitation method used to obtain the confidence intervals
for the model and input parameters in the case study are
described for each parameter in detail in Section 4.3.
3.4. Sensitivity analysis
After eliciting the probability distributions, a sensitivity
analysis is carried out. The model inputs and model parame-
ters are systematically varied, according to their estimated
uncertainty ranges obtained from the elicitation study. Model
simulations were carried out for a timescale relevant to the
particular quality standards. Significant changes in predicted
water quality failures based on the adjustments of single
input and model parameters are thus identified. This
approach was favoured because of the practical level of
computational effort, which meant that the method could be
applied to catchments using complex models. For example,
for the case study sewer network the flow and quality simu-
lation of a 5-month period using Infoworks CS takes approx-
imately 5.5 h to run on a desktop computer (Pentium 4 3.0 GHz
processor and 1 GB of RAM) and generates approximately 7 GB
of model output. The case study is, however, relatively small
as the Infoworks CS model consists only of 505 conduits and
530 nodes. For a larger urban area, the number of nodes and
conduits modelled in a hydrodynamic sewer network model
could easily exceed several thousand. Computation times for
this type of urban drainage model are roughly proportionate
to the number of nodes and conduits modelled.
3.5. Estimation of uncertainty of the ICM outputs
The most influential model inputs and model parameters
identified in the sensitivity analysis are used to estimate the
uncertainty in the number of predicted water quality failures.
A response database is used to describe the variation in
frequency of water quality failures based on the estimated
uncertainty ranges of input and model parameter values. The
elicited probability distributions are then used in Monte Carlo
simulations, thereby interpolating water quality failures from
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 4 3897
the response database instead of running a computationally
expensive deterministic ICM. An estimate of the range and
probability distribution of the frequency of water quality
failure for selectedwater quality determinants is derived from
the Monte Carlo simulation results.
4. Results of the uncertainty analysis
This section describes how the uncertainty analysis method
explained in Section 3 was applied to the case study.
4.1. Mapping of processes and equations in the originalIntegrated Catchment Model
The uncertainty analysis described in this paper only
concerns the Infoworks CS component of the ICM study. Fig. 2
shows diagrammatically how the water and pollutant flows
are represented by a number of empirically calibrated rela-
tionships in the Infoworks CS model. The rainfall runoff
volume and the rainfall runoff routing are calculated for each
of the urban sub-catchments. Of the possible rainfall runoff
and runoff routing model options in Infoworks CS, the New
UK Percentage Runoff model (Packman, 1990) and the Double
Linear Reservoir (Wallingford) model (Sarginson and Nussey,
1982) had been selected. In the flow quality module,
a surface pollutant model calculates the mass inflow of
sediment and pollutants from the sub-catchment surfaces
and from the gully pots into the sewer system. The temporal
build-up of pollutants in the gully pots during dry weather
periods and the flushing of the gully pots during wet weather
events is calculated via a number of empirically calibrated
relationships (Gent et al., 1992). The amount of sediment
building up on the sub-catchment surfaces and the amount of
pollutants attached to the sediment washed off the sub-
catchment surfaces is estimated using empirically calibrated
relationships (Bujon and Herremans, 1990). Dissolved pollut-
ants are represented as entering the system through gully
pots at each node. The amount of suspended sediment and
dissolved and attached pollutants in the domestic waste
water flow is based on historical data reported by Ainger et al.
(1998). The erosion, transport and deposition of suspended
sediment is calculated using the Ackers-White relationships
(Ackers et al., 1996). Sediment can be deposited as well as re-
eroded in the network conduits throughout the model simu-
lation. The sewer network model calculates hydraulic
parameters in each conduit using the conservation of mass
and momentum equations, combined with empirical energy
loss equations. These are solved iteratively throughout the
network. The conduit model estimates the transport of dis-
solved pollutants through the conduits based on simple
advection and the conservation of mass of sediment, pollut-
ants and water.
4.2. Initial sensitivity check
The FIS (FWR, 1998) for a river reach comprise of threshold
concentration values for DO and NH4þ, for 1 or 6 h durations,
with return periods of 1, 3 and 12 months. The flows of BOD
and NH4þ through the urban catchment and sewer system
have therefore been included in this initial sensitivity check,
Fig. 2 shows their potential flow paths.
The original deterministic ICM, or ‘baseline model’ has its
model and input parameters fixed using values obtained
without consideration of uncertainty. Model runs whereby all
pollutant sources except the one being examined are set to
zero, are used to identify the relative mass of pollutants
emitted into the receiving water from different pollutant
sources. Several equations in the urban drainage model have
an element of build-up, decay, erosion or depletion over time.
All sensitivity analyses and uncertainty analyses therefore
need to be carried out using ‘long term’ simulations and not
single storm events. The use of time-series is also advocated
by Rauch et al. (2002), whereas Thorndahl et al. (2008) advo-
cates the use of a large enough number of events in order to
reduce event specific uncertainties. The period used for
analysis of FIS standards in the UK is commonly 10-years of
summer periods. In order to reduce computation times to
a reasonable level the sensitivity analysis for each pollutant
was carried out for a single 5-month summer period obtained
from a representative year from the 10-year rainfall time
series. This representative year was selected by calculating
the cumulative rainfall of each year, and taking the year
where the cumulative rainfall was closest to the average
yearly cumulative rainfall of the 10-year period. Table 1
includes all model inputs and parameters that have not
been studied further because they did not significantly add to
the total amount of pollutants entering the river. All the
remaining processes were included in the elicitation process,
Table 1 also includes the processes rejected during this
elicitation process.
4.3. Elicitation of probability distributions
The elicitation process was undertaken by the paper authors,
the method was structurally based and distributions of
selected input and model parameters were estimated based
mainly on expert knowledge as field datawas very limited. The
paper authors are academic experts in urban drainage quality
processes, they initially carried out the elicitation amongst
themselves and the results were subsequently discussed with
the urban drainage modelling experts at the sewer operator.
After discussion with the paper authors and the modelling
experts at the sewer operator, a consensuswas reached so that
for each elicited model input and model parameter a single
probability distribution was selected. Published data and
information available on the development of the equations
incorporated in Infoworks CS was gathered, together with the
limited amount of data available from the actual case study
catchment. The model inputs and model parameters were
assumed to be normally distributed unless the available data
or literature sources indicated a lognormal distribution. Based
on the limited amount of data as well as the information
gathered, 95% confidence intervals (i.e. 2.5 and 97.5 percen-
tiles) of model input and model parameter values had been
estimated. These value rangeswere thendiscussedwithurban
drainagemodelling experts at the sewer operator, after which
adjustment had to be made, if necessary.
For example, the width of the distribution for soil moisture
depth had to be adjusted because of the modelling experts’
Fig. 2 e Schematic overview of the modelled water and pollutant flows in the sewer network model used in the Integrated
Catchment Modelling study.
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 43898
uneasewith thehighest soilmoisture depth found in literature
in comparison with their experience of modelling catchments
in the region. The distribution of domestic waste water flow
particle size also had to be adjusted because of the boundaries
of applicability of the sediment transport equation.
Based on the 95% confidence intervals of input parameter
and model parameter values, probability distributions were
derived based on the assumption that the 95% confidence
interval can be approximated by þ/� twice the standard
deviation (s). Using this method, statistical distributions have
been compiled for the relevant model inputs and model
parameters identified by the initial sensitivity study. Table 2
summarises these inputs and parameters and their elicited
probability distributions. After the initial sensitivity check, the
following processes remained to be investigated: catchment
surface runoff (water) and wash-off (sediments and soluble
pollutants); surface pollutant build-up and decay; domestic
waste water flow; erosion, deposition and transport of sedi-
ment through the sewer system;water flow through the sewer
system. The process of estimating 95% confidence intervals of
the input and model parameters associated with these
processes is described in detail below.
It must be appreciated that the elicitation process is not
exact and requires the use of judgement. Previous work by
O’Hagan (1998) has indicated that estimating 95% confidence
intervals can lead to an underestimation of the extremes of
the range. Consideration is also required to ensure the
distributions derived from an elicitation process do not
contain physically impossible values, so a good knowledge of
underlying physical processes is required. This is straight
forward when parameter have a physical meaning, such as
the input parameter particle size, but for several model
parameters these are purely empirical and so do not have
a direct physical meaning. A good understanding of the
behaviour of the system is then required to ensure that the
model parameters cannot take values that would predict
behaviour that is physically impossible. In some cases there
was still a very small probability of physically impossible
negative values being drawn, when this occurred during the
Monte Carlo simulations these values were changed to zero.
4.3.1. Catchment surface runoffThe original data used to derive the New UK Runoff Model are
described in Packman (1990). The soil moisture depth
parameter had originally been used for calibrating the NewUK
Runoff Model, which led to a soil moisture depth of 200 mm
being recommended as a default value in the UK. This is the
default value currently incorporated in Infoworks CS. Seven
catchments had been studied by Packman (1990), and the soil
moisture depths ranged from 541 to 96 mm. It was assumed
that the data would be normally distributed, the mean of the
data is 236 mm and the standard deviation 144 mm. When
discussing the Packman (1990) data with the engineering
modellers, the highest moisture depth value of 541 mm was
Table 1 e Sewer network model parameters & modelinputs rejected during initial mapping of processes,initial sensitivity checks and the elicitation process.
Model parameters/inputs Reason for rejection aftermapping & initial sensitivity
checks
sub-catchment slope Model output not sensitive to
a 50% increase or decrease
sub-catchment areas, runoff
coefficients, sewer network
geometry (Conduit lengths,
pipe gradients, diameters,
weir heights, orifices,
manhole dimensions)
Clemens and Von der Heide
(1999) indicated that reasonable
levels of uncertainty in sub-
catchment areas and a sewer
network geometry database
would lead to up to 10%
variations in predicted flow
quantity. The volume of CSO
spills was more sensitive to CSO
weir heights. Data or expert
knowledge on typical variation in
CSO weir heights was not
available.
Soil type (‘New UK’ runoff
model)
Packman (1990) indicated that
only the soil moisture depth had
been used to calibrate the ‘New
UK’ runoff equation
Rainfall, rainfall intensity,
duration of dry periods
Future expected rainfall had been
derived using the ‘STORMPAC’
software package, estimating the
associated uncertainty fell
outside the scope of this study.
Rainfall erosion equation
coefficients
Artina et al. (2007) reported these
to be very sensitive, but no expert
knowledge on the development
of the equation and its
coefficients was found.
Coefficients in the
Ackers-White equations
Were hard-coded and hence
could not be adjusted
Trade flow quantity
and quality
Contribution of traders was
negligible in the case study
Numerical parameters
(number of computational
nodes with each conduit
and number of subdivision
of each timestep)
According to Bouteligier et al.
(2005), numerical dispersion can
be a problem in Infoworks CS.
The number of computational
nodes had been increased from
default 5 to the maximum
possible number, 40. This gave
a difference of �15% to þ20%
spilled pollutant mass, but not in
all CSOs
Gully pot build-up equation
coefficients
As reported by Gent et al. (1992)
outputs of these equations were
not very sensitive to any of the
coefficients
Table 2 e Infoworks CS model parameters & modelinputs, their expected uncertainty ranges and assumedstatistical distributions (m[mean and s[ standarddeviation).
Model parameters/inputs Infoworks CS taken into accountfor sensitivity analysis
Soil moisture depth
(‘New UK’)
Normal, m¼ 200 mm, s¼ 75 mm
Hydraulic roughness of
sewer conduits
Normal, s¼ 25% of m
Potency factor (‘Kpn’)
equation coefficients
For BOD C1: Lognormal, Logn_m¼�1.3,
Logn_s¼ 0.246
Particle size of suspended
and deposited particles.
(‘Particle size’)
Domestic waste water flow particles:
Normal, m¼ 0.1 mm, s ¼ 0.02 mm
Surface sediment particles: Lognormal,
m¼ 1.1 mm, s¼ 0.45 mm
Dry weather flow pattern,
quantity and quality
Dry weather flow pollutant
concentration: Normal, s¼ 5% of m
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 4 3899
met with significant unease. After discussion, it had been
decided that a more reasonable 95% confidence interval for
describing soil moisture depth values in the UK would be
between 50 mm and 350 mm, regardless of the soil type. This
adjustment addressed the unease of the modellers.
4.3.2. Surface sediment build-up and decay factorsSurface sediment build-up and decay factors express the
build-up and decay rate of sediments on the catchment
surface. According to Ellis (1986) and Ashley et al. (2004), the
surface sediment build-up parameter (Ps) can vary between
2 kg/ha/day and 10 kg/ha/day in residential areas. Reliable
literature sources or experts on the rate of decay were not
found, hence the uncertainty in the decay rate could not be
elucidated.
4.3.3. Surface sediment erosion and wash-off process,attachment of pollutants to surface sediment (potency factorequations)Wash-off is the process of erosion and transportation of
sediments over the catchment surfacewith the rainfall runoff.
Model parameter uncertainty is associated with the coeffi-
cients in the potency factor equations and the rainfall erosion
equation. Potency factors are a measure of the ratio with
which BOD is attached to surface sediments. Each surface
sediment potency factor is described by the rainfall intensity
and four calibration coefficients, in the form of: Potency
factor¼C1(Intensity�C2)C3þC4. This equation was derived
from field data by Bujon and Herremans (1990), this data only
consisted of two rainfall events measured in a single catch-
ment in France. Varying C1 between 0.56 and 0.14 creates an
uncertainty range enveloping all potency factors derived from
the limited field data described in Bujon and Herremans
(1990). The limited amount of field data did suggest
a lognormal distribution would be more suitable. Considering
the uncertainty involved in using two events measured in
France to describe the attachment of pollutants to sediment
running off the surfaces in the UK, it was deemed sufficient to
just vary C1 to get an estimate of the uncertainty in the model
output caused by the spread in the Bujon and Herremans
(1990) field data.
4.3.4. Hydraulic flows through the sewer systemRainfall is a considerable source of model input uncertainty
associated with the amount of flow running off the urban
catchment surface, and hence entering the urban drainage
system. Future rainfall was, however, predicted by STORM-
PAC, the uncertainty in STORMPAC model output was not
included in the scope of this study. Uncertainty associated
with the catchment runoff and routing equations have been
discussed in Section 4.3.1. Model input uncertainty can
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 43900
furthermore be associated with the hydraulic roughness of
the sewer conduits as well as the sewer system geometry.
Hydraulic roughness changes with pipe wall condition, sedi-
mentation and water level (Wotherspoon, 1994). Uncertainty
in the estimation of the hydraulic roughness of sewer pipes
has been studied in Schellart (2007); the findings of this study
suggested that the hydraulic roughness can vary between
þ/�50% of the estimated hydraulic roughness values in well
calibrated hydraulic sewer network models. As the above-
mentioned studies suggested a symmetrical distribution of
data, a normal distribution was selected with a 95% confi-
dence interval ofþ/�50% of the hydraulic roughness values in
the case study sewer network model.
4.3.5. Dry weather flow pattern and pollutantsThe baseline model, which did not incorporate any consider-
ation of uncertainty, used a standard dry weather flow and
pollutant pattern derived from a large field data set collected
in the UK, see Ainger et al. (1998). A limited amount of field
data available from the present field study site suggested that
the dry weather flow pollution concentrations varied between
þ/�10% from the default average dry weather flow pollution
concentrations that had been derived fromAinger et al. (1998).
A normal distribution was selected with a 95% confidence
interval of þ/�10% of the dry weather flow pollution concen-
trations in the case study sewer network model.
4.3.6. SedimentsSewer sediments can originate from the catchment surface,
domestic wastewater flows, infiltratingwater and trade flows.
Model input uncertainty can be associated with the particle
size and density of the sediment and model parameter
uncertainty can be associated with the coefficients within the
Ackers-White sediment transport equation. An overview of
various different literature sources and shows that the
particle size and density in domestic waste water flow can
vary widely in both time and space e.g. Chebbo et al. (1990),
Verbanck et al. (1990) and Wotherspoon (1994). Taking into
account the boundaries for use of the Ackers-White equa-
tions, it was decided to select a normal distributionwith a 95%
confidence interval between 0.05 mm and 0.15 mm for the
domestic waste water flow particle size, with a density of
2200 kg/m3. For the surface particles, the published data sug-
gested a lognormal distribution would be suitable. A 95%
confidence interval between 0.2 mm and 2 mm, and a density
of 2600 kg/m3 was selected. It has to be noted that the limited
amount of published data on domestic waste water flow
particle size and density indicates that most of the particle
size/density combinations lay outside the boundaries of the
Ackers-White model. The model parameter uncertainty
associated with Ackers-White has been studied in Schellart
(2007) and this uncertainty can be significant. However, as
the Ackers-White coefficients cannot be adjusted in Infoworks
CS (v 7.0) a study into the network wide effect of this model
parameter uncertainty could not be carried out.
4.4. Sensitivity analysis e case study
All sensitivity analysis model runs have been compared with
the predictions obtained from the ‘baseline’ deterministic
ICM, using the same 5-months rainfall as the ‘baseline’ model
run. For the sensitivity analysis, all model inputs/parameters
listed in Table 2 have been changed individually according to
their estimated 2.5 and 97.5 percentile values, whilst keeping
all other model parameters set at their default values and
model inputs at their baseline values.
The results are shown in Fig. 3, for clarity, the scatter
graphs only show the more sensitive model inputs/parame-
ters (soil moisture depth, dry weather flow for NH4þ and
domestic waste water particle size and soil moisture depth for
BOD). The emitted masses of pollutants predicted by the
model runs have been cumulated over 6-h periods, in order to
ensure that the sensitivity of Infoworks CS output is compared
over the same time-frame as the FIS standards. The highest
relative as well as actual over- and under prediction of BOD
emitted from all outfalls, when compared with the baseline
run, was caused by changing the soilmoisture depth to 50 mm
and the domestic waste water particle size to 0.1 mm. The
amount of NH4þ emitted from the sewer system is most
sensitive to the soil moisture depth parameter, and, as
expected, a 10% increase in the amount of NH4þ in the
domestic waste water flow leads to a 10% increase in NH4þ
spilled.
4.5. Estimation of the ICM output uncertainty
Following the statistical distributions for the most sensitive
parameters, simulations were conducted using the full ICM
(Fig. 1), in order to create the response database.
The full ICM procedure has been run for a range of possible
combinations of soil moisture depth (50e350 mm in steps of
50 mm) and particle size (0.05e0.15 mm in steps of 0.025 mm),
i.e. 25 model runs in total. For all these simulations, the same
5-month rainfall series was used. For each quality standard
a separate response database can then be populated with the
number of predicted failures; Table 3 shows one example
response database. During a Monte Carlo analysis, 10,000
random draws were made from the elicited probability
distributions of particle size and soil moisture depth. The
accompanying incidents of water quality failures are linearly
interpolated from the response database (taking into account
that the number of failures is an integer value), removing the
need to run the original ICM an additional 10,000 times. The
actual probability of failure of a standard can then be derived
by comparing the 10,000 Monte Carlo simulation results with
the surface water quality standards, i.e. the allowed number
of failures in a given period of time. If nf is the number of
model runs where the system fails, and n is the total number
of runs then Pf¼nf/n, where Pf is the probability of system
failure.
Fig. 4 shows the probability of failure of the 6-h, 1, 3 and 12-
months return period cyprinid FIS standards for DO and NH4þ,
due to uncertainty in the particle size and the soil moisture
depth of the Infoworks CS model. In the baseline model run,
26 failures of the DO 6-h standard with a 12-month return
period were predicted. When uncertainty in particle size and
soil moisture is taken into account, the number of failures can
vary between 22 and 31 (Fig. 4). Thus, the uncertainty in
particle size and soil moisture depth has caused a possible
range of 35% variation in the number of predicted failures.
Fig. 3 e Relative over- or under prediction in the mass of Biochemical Oxygen Demand (BOD) and unionised ammonia (NH4D)
released from all Combined Sewer Overflows compared with model predictions obtained without considering uncertainty,
Section 4.4.
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 4 3901
Fig. 4 also indicates that estimated probability of the number
of failures is generally not normally distributed. The range of
variation in the predicted failures of the other FIS standards
due to uncertainty in soil moisture depth and particle size is
summarised in Table 4. Application of the uncertainty esti-
mation method indicated that due to uncertainty in the soil
moisture depth and the domestic waste water flow particle
size in the sewer model, there was a potential uncertainty
range in the number of predicted water quality failures of
between 45% and 32% respectively, for DO and NH4þ standard
failures in the CSO spills.
5. Discussion
A reduction in computer simulation time of approximately
two orders of magnitude was made by using response data-
bases. Creation of the response databases took 6 days of run
time on one desktop computer (Pentium 4 3.0 GHz processor
and 1 GB of RAM), compared with an estimated run time of
over 6 years if the full ICM were to be run 10,000 times on
a similar desktop computer. It has to be noted that the Info-
works CS model runs relatively slowly when water quality
prediction is included; 5.5 h for a single representative
5-month simulation period for the case study catchment.
Computational constraints have been noted even by
researchers using simpler and faster running conceptual
Table 3 e Example Response Database (for the number offailures of the 6 h DO cyprinid FIS standard with a returnperiod of 12 months, for the case study).
Soil moisture depth (mm)
50 125 200 275 350
Particle size (mm) 0.05 22 27 27 31 30
0.075 22 26 26 31 30
0.1 22 26 27 31 30
0.125 22 26 27 31 30
0.15 22 25 27 31 30
models (Haydon and Deletic, 2009) as well as authors using
hydrodynamic sewer models without water quality processes
(Thorndahl et al., 2008). The time saving that can be made by
using a response database is dependent on the range of
uncertain input and model parameters included, as the
number of simulations necessary to populate the response
databases increases with the number of parameters included
in the response database. A second issue is that only model
outputs that can be expressed as a discrete result can be
included in the response database; the database cannot be
populated with time series. Water quality standards are,
however, usually expressed in terms of concen-
trationedurationefrequency and thus ‘standard’ failures can
be captured in a single number. Finally, there is also a level of
uncertainty involved in using a response database, instead of
the full ICM in the Monte Carlo simulations, Schellart et al.
(2010) demonstrated that the level of uncertainty inherent in
using a well constructed response database was significantly
less than the predicted modelling uncertainty.
The comprehensive uncertainty mapping procedure used
from the outset of the analysis also included all other poten-
tial sources of uncertainty, and explicitly mentions those that
have been ignored/could not be taken into account (Table 1). It
therefore provides key information for application to similar
future uncertainty studies on other catchments, as well as
future deterministic ICM studies. Conceptualisation of the
catchment as a series of reservoir models is common in
a number of studies (e.g. Freni et al., 2008;Mannina et al., 2006;
Willems, 2008), however, this approach was not used here. A
network model approach, where networks of sewer conduits
were modelled was selected instead, in order that the spatial
and temporal differences in sediment and pollutant transport
capacity, as well as deposition and erosion between different
sewer sections could be taken into account. The case study
system was relatively steep, with an average gradient of
0.0218 for all themodelled conduits. In sewer systems that are
less steep, uncertainty due to sediment erosion, transport and
deposition can be far more influential, Schellart et al. (2008).
The expert elicitation process used requires careful appli-
cation, if it is to provide a reasonable estimate of output
uncertainty (Garthwaite et al., 2005; Kadane and Wolfson,
Fig. 4 e aef. Histograms of the prediction of the number of Fundamental Intermittent Standard failures due to uncertainty in
soil moisture depth and particle size.
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 43902
1998; O’Hagan, 1998). During the case study described in this
paper several difficulties in the elicitation process were
encountered. The development of some empirically calibrated
quality equations incorporated in the sewer network model
used were not documented. Hence little expert knowledge
could be found on the development and original calibration of
themodel parameters and their probability distributions could
not be elicited or included in the sensitivity analysis. There
were also limitations in the sediment transport equations
incorporated in the sewer network model. Literature on sedi-
ment particle size in domestic waste water flow (e.g. Chebbo
et al., 1990; Verbanck et al., 1990; Wotherspoon, 1994) indi-
cates that the elicited probability distribution for domestic
waste water flow particle size is almost certainly too narrow.
However, a wider distribution could not be selected, as the
sediment transport equations are only validwithin a restricted
Table 4e Range of variation in the number of 6-h cyprinidFIS standard failures predicted when uncertainty in soilmoisture depth and particle size is taken into account,compared with the number of failures predicted with thebaseline model.
FISstandard
Range ofvariation
FIS standard forNH4þ
(return period)
Range ofvariation
DO 1-month
rp
�10% to þ4% 1-month �11% to þ3%
DO 3-month
rp
�8% to þ5% 3-month �12% to þ6%
DO 12-
month rp
�15% to þ19% 12-month �37% to þ12%
particle size/particle density combination range, an issue first
reported by Bouteligier et al. (2002). Another issue arose when
examining the soil moisture depths reported in Packman
(1990) with the modelling experts at the sewer operator. The
highest moisture depth value of 541 mm was met with
significant unease, and after discussion with the modelling
experts a narrower distribution was selected. It is difficult to
test the judgement of these experts as there is little reliable UK
urban soil moisture depth data. Based on the experience of
elicitation described in the literature, as well as the specific
issues encountered in this case study it was believed that the
widths of the elicited probability distributions derived for this
case study are under-estimates. The uncertainty ranges found
in the ICM water quality outputs are therefore expected to be
under-estimates, not only due to limitations in the elicitation
process, but also because they only account for uncertainty
traced back to the Infoworks CS model. Uncertainty related to
the rainfall generator, the hydrological rainfall runoff model
and the river pollutant transport and transformation model
have not been taken into account.
6. Conclusions
This study has identified and reviewed all the processes in the
sewer flow quantity and quality model component of an
integrated catchment modelling study. For these processes,
an elicitation process was used to estimate appropriate
probability distributions of model inputs and parameters to
quantify their uncertainty. This information was then used to
estimate the impact of these uncertainties on the levels of
uncertainty associated with the number of water quality
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 4 3903
failures predicted when themethodwas applied to a small UK
catchment for which a purely deterministic ICM was already
available.
The following conclusions were drawn:
� Expert elicitation can be used to estimate the level of
uncertainty in model inputs and parameters for urban
drainage systems even when field data is limited.
� By combining descriptions of uncertainty from selected
model and input parameters an estimate of the uncertainty
in the predicted number of water quality failures can be
obtained in an ICM study.
� A Monte Carlo approach whereby ICM predictions are
interpolated from a response databases can be used to
reduce significantly computation times and so make the
estimation of the uncertainty in ICM predictions possible
when using complex deterministic models of real
catchments.
� The sensitivity and uncertainty analysis method described
in this paper can be readily adapted for use with other
complex deterministic commercial software packages used
to model aspects of real catchments in ICM studies.
� Analysis of the uncertainty levels associated with predicted
values can be used to optimise resources spent on data
collection and the implementation of any solution. It is
expected that the levels of uncertainty in ICM outputs found
in this study are under-estimates. This is because only the
uncertainty in the sewer networkmodel has been taken into
account, several model and input parameters could not be
elicited and some of the elicited probability distributions
were expected to be too narrow. With the levels of uncer-
tainty reported any solutions designed based on this
deterministic ICM are likely to be significantly over- or
under-dimensioned. This level of uncertainty therefore has
considerable significance in a time of increasing uncertainty
in external inputs such as rainfall and runoff and the ability
of receiving watercourses to assimilate polluting inputs due
to a changing climate.
Acknowledgement
This study has been funded by Yorkshire Water Services
(YWS) Ltd. The views expressed in this paper are, however,
not necessarily the views of YWS Ltd. The authors would like
to thank YWS Ltd. andMWHUK Ltd. and their staff for sharing
their experience and knowledge thereby providing valuable
input to this project.
r e f e r e n c e s
Ackers, J.C., Butler, D., May, R.W.P., 1996. Design of Sewers toControl Sediment Problems. Report 141. Construction IndustryResearch and Information Association, London, UK.
Ainger, C.M., Armstrong, R.J., Butler, D., 1998. Dry Weather Flowin Sewers. Report 177. Construction Industry Research andInformation Association, London, UK.
Artina, S., Bolognesi, A., Liserra, T., Maglionico, M., 2007.Simulation of a storm sewer network in industrial area:
comparison between models calibrated through experimentaldata. Environ. Model. Software 22, 1221e1228.
Ashley, R.M., Bertrand-Krajewski, J.-L., Hvitved-Jacobsen, T.,Verbanck, M. (Eds.), 2004. Solids in Sewers. Scientific andTechnical Report No. 14. IWA Publishing, London, UK.
Ashley, R., Bertrand-Krajewski, J.-L., Hvitved-Jacobsen, T., 2005.Sewer solidsd20 years of investigation. Water Sci. Technol. 52(3), 73e84.
Benedetti, L., Bixio, D., Vanrolleghem, P.A., 2005. Assessment ofWWTP design and upgrade options: balancing costs and risksof standards’ exceedance. In: Proc. 10th Int. Conf. on UrbanDrainage, 21e26 August, Copenhagen, Denmark.
Bertrand-Krajewski, J.-L., 2006. Influence of field data sets oncalibration and verification of stormwater pollutant models. In:Proc. 7thConf.UrbanDrainageModelling,Melbourne,Australia.
Bertrand-Krajewski, J.-L., Bardin, J.-P., 2002. Evaluation ofuncertainties in urban hydrology: application to volumes andpollutant loads in a storage and settling tank. Water Sci.Technol. 45 (4e5), 437e444.
Beven, K.J., Binley, A.M., 1992. The future of distributed models e
model calibration and uncertainty prediction. Hydrol. Process.6 (3), 279e298.
Bouteligier, R., Vaes, G., Berlamont, J., 2002. In sewer sedimentand pollutant transport models. In: Proceedings 3rd SewerProcesses and Networks, Paris, France, pp. 47e54.
Bouteligier, R., Vaes, G., Berlamont, J., Flamink, C., Langeveld, J.G.,Clemens, F., 2005. Advectionedispersion modelling tools:what about numerical dispersion? Water Sci. Technol. 52 (3),19e27.
Box, G.E.P., 1954. The exploration and exploitation of responsesurfaces: some general considerations and examples.Biometrics 10, 16e60.
Bujon, G., Herremans, L., 1990. Modele de prevision des debits etdes flux polluants en reseaux d’assainissement par temps depluie Calage et validation. La Houille Blanche, No 2.
Chebbo, G., Musquere, P., Milisic, V., Bachoc, A., 1990.Characterization of solids transferred into sewer trunksduring wet weather. Water Sci. Technol. 22 (10/11), 231e238.
Clemens, F.H.L.R., Von der Heide, W., 1999. Effect of geometricalerrors in hydrodynamic calculation in urban drainage. In:Proceedings of the 8th ICUSD, Aug. 1999, Sydney, Australia,pp. 955e963.
Dahal, M.R., Petry, B., van Gelder, P.H.A.J.M., Gupta, S., Vrijling, J.K.,2005. Reliability analysis of flood defences using importancesampling and response database with probabilistic loops inlarge hydraulic models. In : Proceedings ISSH e StochasticHydraulics, 23e24 May 2005, Nijmegen, The Netherlands.
DHI (Website checked February 2009). http://www.dhigroup.com/Software/WaterResources/MIKEBASIN/Details/RainfallRunoff.aspx.
Ellis, J.B., 1986. Pollutional aspects of urban runoff. In: Torna, H.C.,Marsalek, J., Desbordes, M. (Eds.), NATO ISI Series. UrbanRunoff Pollution, vol. G10. Springer Verlag, Berlin, Heidelberg.
Freni, G., Mannina, G., Viviani, G., 2008. Uncertainty in urbanstormwater quality modelling: the effect of acceptabilitythreshold in the GLUEmethodology. Water Res. 42, 2061e2072.
FWR (Foundation for Water Research), 1998. Manual, UrbanPollution Management, second ed. (CD Rom). October 1998,FR/CL0009.
Garthwaite, P.H., Kadane, J.B., O’Hagan, A., 2005. Statisticalmethods for eliciting probability distributions. J. Am. Stat.Assoc. 100, 470.
Gent, R.J., Crabtree, R.W., Osborne, M.P., 1992. MOSQITO 2Testing. Foundation for Water Research Report No. FR0311.
Harremoes, P., Madsen, H., 1999. Fiction and reality in themodelling world e balance between simplicity andcomplexity, calibration and identifiability, verification andfalsification. Water Sci. Technol 39 (9), 1e8.
wat e r r e s e a r c h 4 4 ( 2 0 1 0 ) 3 8 9 3e3 9 0 43904
Haydon, S., Deletic, A., 2009.Model output uncertainty of a coupledpathogen indicator-hydrologic catchment model due to inputdata uncertainty. Environ. Model. Software 24, 322e328.
Kadane, J.B., Wolfson, L.J., 1998. Experiences in elicitation. TheStatistician 47 (1), 3e19.
Kanso, A., Gromaire, M.-C., Gaume, E., Tassin, B., Chebbo, G.,2003. Bayesian approach for the calibration of models:application to an urban stormwater pollution model. WaterSci. Technol. 47 (4), 77e84.
Korving, H., 2004. Probabilistic assessment of the performance ofcombined sewer systems. PhD thesis, Technische UniversiteitDelft, Delft, The Netherlands.
Khuri, A.I., Cornell, J.A., 1987. Response Surfaces: Design andAnalyses. Marcel and Dekker, New York.
Mannina, G., Freni, G., Viviani, S., Sægrov, S., Hafskjold, L.S., 2006.Integrated urban water modelling with uncertainty analysis.Water Sci. Technol. 54 (6e7), 379e386.
McCarthy, D.T., Deletic, A., Mitchell, V.G., Fletcher, T.D., Diaper, C.,2008. Uncertainties in stormwater E. coli levels. Water Res. 42,1812e1824.
Mourad, M., Bertrand-Krajewski, J.-L., Chebbo, G., 2005.Sensitivity to experimental data of pollutant site meanconcentration in stormwater runoff. Water Sci. Technol. 51 (2),155e162.
Muschalla, D., Schutze, M., Schroeder, K., Bach, M., Blumensaat, F.,Klepiszewski, K., Pabst, M., Pressl, A., Schindler, N., Wiese, J.,Gruber, G., 2008. The HSG guideline document for modellingintegrated urban wastewater systems. In: Proceedings 11thInternational Conference on Urban Drainage, Edinburgh,United Kingdom.
O’Hagan, A., 1998. Eliciting expert beliefs in substantial practicalapplications. The Statistician 47 (1), 21e35.
Osborne, M., Lau, T., 2003. Water quality modelling e Simpol,Simon and Infoworks. In: WaPUG Autumn Meeting, Blackpool,United Kingdom. http://www.ciwem.org/groups/wapug/.
Packman, J., 1990. New Hydrology model. In: WaPUG SpringMeeting 1990. http://www.ciwem.org/groups/wapug/.
Pappenberger, F.,Harvey,H., Beven,K.,Hall, J.,Meadowcroft, I., 2006.Decision tree for choosing anuncertainty analysismethodology:a wiki experiment. Hydrol. Process. 20 (17), 3793e3798.
Priestley, M., Barker, C., 2006. Integrated catchment modelling forfreshwater fisheries directive compliance. In: Scottish WaPUGSpring Meeting. http://www.ciwem.org/groups/wapug/.
Rauch, W., Aalderink, H., Krebs, P., Schilling, W., Vanrolleghem, P., 1998. Requirements for integrated wastewater models e
driven by receiving water objectives. Water Sci. Technol. 38(11), 97e104.
Rauch, W., Bertrand-Krajewski, J.-L., Krebs, P., Mark, O.,Schilling, W., Schutze, M., Vanrolleghem, P.A., 2002.Deterministic modelling of integrated urban drainagesystems. Water Sci. Technol. 45 (3), 81e94.
Rousseau, D., Verdonck, F., Moerman, O., Thoeye, C., Meirlaen, J.,Vanrolleghem, P.A., 2001. Development of a risk assessmentbased technique for design/retrofitting of WWTPs. Water Sci.Technol. 43 (7), 287e294.
Sarginson, E.J., Nussey, B.B., 1982. The explicit computation ofurban runoff. In: Featherstone, R.E., James, A. (Eds.), UrbanDrainage Systems: Proceedings of the First InternationalSeminar, Southampton, England, September 1982, ISBN0273085964.
Schellart, A.N.A., 2007. Analysis of uncertainty in the sewersediment transport predictions used for sewer managementpurposes. PhD thesis, University of Sheffield, United Kingdom.
Schellart, A.N.A., Buijs, F.A., Tait, S.J., Ashley, R.M., 2008.Estimation of uncertainty in long term combined sewersediment behaviour predictions, a UK case study. Water Sci.Technol. 57 (9), 1405e1412.
Schellart, A., Tait, S., Ashley, R.M., 2010. Estimationofuncertainty inlong termsewersedimentbuild-uppredictions,usinga responsedatabase. Journal of Hydraulic Engineering 136 (7), http://dx.doi.org/10.1061/(ASCE)HY.1943-7900.0000193.
Slijkhuis, K.A.H., Van Gelder, P.H.A.J.M., Vrijling, J.K.,Vrouwenvelder, A.C.W.M., 1999. On the lack of information inhydraulic engineeringmodels. In: Schuller, G.I., Kafka, P. (Eds.),Safety and Reliability. Proc. of Esrel ’99 e 10th EuropeanConference on Safety and Reliability. Munich-Garching,Germany, pp. 713e718.
Thorndahl, S., Beven, K.J., Jensen, J.B., Schaapur-Jensen, K., 2008.Event based uncertainty assessment in urban drainagemodelling, applying the GLUE methodology. J. Hydrol. 357,421e437.
Verbanck, M., Vanderborght, J.P., Wollast, P., 1990. Sizedistributions of suspended particles in combined sewersduring dry and wet weather. In: Proc. 5th Int. Conf. on UrbanStorm Drainage, Osaka, Japan, pp. 891e896.
WaPUG, 2002. Code of Practice for the Hydraulic Modelling ofSewer Systems. http://www.ciwem.org/groups/wapug/.
Willems, P., 2008. Quantification and relative comparison ofdifferent types of uncertainties in sewer water qualitymodelling. Water Res. 42, 3539e3551.
Willems, P., 2010. Parsimonious model for combined seweroverflow pollution. J. Environ. Eng 136 (3), 316e325.
Willems, P., Berlamont, J., 1999. Probabilistic modelling of sewersystem overflow emissions. Water Sci. Technol. 39 (9), 47e54.
Wotherspoon, D.J.J., 1994. The movement of cohesive sediment ina large combined sewer. PhD thesis, University of Abertay,Dundee, Scotland.
WRc (website checked, February 2009). http://www.wrcplc.co.uk/stormpac/asp/about.asp.
Wynne, B., 1992. Uncertainty and environmental learning.Reconceiving science and policy in the preventive paradigm.Global Environ. Change 2 (2), 111e127.