aged sorption: reliability of estimated model parameters · calculation of cv _25,true-param...
Post on 10-Aug-2020
2 Views
Preview:
TRANSCRIPT
Aged sorption: Reliability of
estimated model parameters
Mechteld ter Horst, Jos Boesten (Alterra)Wendy van Beinum, Sabine Beulke (FERA)
Outline
Goal
Method
Results parameter estimation
Criteria for an acceptable fit
Conclusions
Goal
Identify for which cases parameter estimation is not possible
Problem with complex models: fit large number of model parameters
Therefore derivation of reliable parameters is challenging
Therefore important to test in which cases the model fitting doesn’t work (e.g. compounds with weak or strong sorption or with fast or slow degradation).
Give guidance on:
Parameter estimation (weight factor, no of sampling times)
Evaluate the results of the parameter estimation
Method: overview
Generate hypothetical datasets with Monte Carlo simulations using random errors as observed in FERA experiments
Parameter estimation (PEST) for each parameter combination using hypothetical datasets as measurements
Calculate for each parameter combination for fNE kd the CV_25,true-param
(coefficient of variation)
Identify for which cases parameter estimation is not possible (contour plots)
1
2
3
4
5Use parameter estimation output to find method on how to judge if the outcome of the parameter estimation is acceptable
Method: Generate 9450 datasets with Monte Carlo simulations using
random errors from FERA experiments
378 parameter combinations (see next slide)
25 hypothetical datasets per parameter combination containing per dataset:
measurements of mass and concentration on 9 sampling times
3 replicate measurements per sampling time
These datasets are
used as measurements
in the parameter
estimation
process.
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0 20 40 60 80 100 120
time (days)
co
nc
en
tra
tio
n (
ug
.mL
-1)
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0 20 40 60 80 100 120
time (days)
co
nc
en
trati
on
(u
g.m
L-1
)
true values
replicate set 1
replicate set 2
replicate set 3
Example hypothetical dataset incl. random errors
Method: Generate 9450 datasets with Monte Carlo simulations using
random errors from FERA experiments
378 parameter combinations (6*7*3*3)
DegT50
(d)
10 20 40 60 80 100
KF,eq (L/kg)
0.1 0.32 1 3.2 5.5 7.5 10
kd
(d-1)
0.005 0.01 0.03
fNE
(-)
0.3 0.6 1.2
(6)
(7)
(3)
(3)
Freundlich exponent, N = 0.9 for all parameter combinations
Method: parameter estimation
PEST (Doherty, 2006): minimise the difference between simulated and measured values.
Sum of the squared residuals used as criterion function (object function Φ):
weighting of data needed: mass order of magnitude larger than concentration
weighting data mass and concentration got equal importance
2
1
m
i
ii rw
ri is the residual (difference between simulated and measured value corresponding to measurement i ),
wi is the weighting factor
m is the sum of numbers of measurements of mass and concentration
Method: parameter estimation
Estimated parameters: fNE, kd, Kom,eq, DegT50 and initial mass Pest runs for 8 sets of starting values
For each parameter combination (378), hypothetical dataset (25) and set of starting values (8) a PEST_PEARLNEQ run is performed (75600 runs).
Example PEST output
OPTIMISATION RESULTS
Parameters ----->
Parameter Estimated 95% percent confidence limits
value lower limit upper limit
fsne 0.211204 8.766462E-02 0.334744
crd 6.662777E-03 1.876465E-04 1.313791E-02
dt50 19.7934 19.0784 20.5084
masini 42.1954 40.4723 43.9185
komeq 280.040 263.763 296.317
Objective function ----->
Sum of squared weighted residuals (ie phi) = 0.3111
Method: calculation of CV_25,true-param (coefficient of variation)
Calculation of CV_25,true-param fNE and kd for each parameter combination (378)
1. Select from fits with 8 sets of starting values the best fit (lowest Φ)
2. Calculate from 25 fits (25 hypothetical datasets) the CV25,true-param according:
Note: kd as an example s = standard deviation
Small CVs indicate a small variability between the 25 fitted parameter values and a small deviation from the true value
25
...25
1
2
i
kdtruekdfitted.
skdtrue
sparamtrueCV
....25
Results : CV25,true-param of fNE on contour
0.005
0.01
0.03
fNE
kd 0.3 0.6 1.2
Results: CV25,true-param kd on contour
0.005
0.01
0.03
fNE
kd
0.3 0.6 1.2
Results parameter estimationResults visible from contour diagrams
i. Decreasing CV for increasing kd and fNE values
ii. Decreasing CV for increasing KF,EQ values, (except for DegT50 = 10 days and low values of kd and fNE)
iii. Increasing CV for increasing DegT50 values
For explanations see report
kd 0.005, fNE 0.3 large CV
kd 0.03, fNE
1.2 small CV
Larger KF,eq
smaller CV
Larger DegT50 larger CV
Criteria for an acceptable fit: Aim of analysis
Goal:
Test if the confidence interval of single fit can be used as an indication for an accurate parameter estimation (fitted parameter value close to true value).
Method:
compare confidence interval with difference between true and fitted value
Advantage of hypothetical dataset:
we know the true values and can therefore test how well the fitted values come close to the true values
Criteria for an acceptable fit: uncertainty of fitted parameter
Criterion 1: uncertainty measure: CV1,PEST conf.intv
large uncertainty large confidence interval
the 95% confidence interval (PEST output) of 1 fitted parameter of 1 fit (hypothetical dataset) is used to calculate the CV1,PEST conf.intv (RSE in report)
Large CV1,PEST conf.intv indicates large uncertainty
Uncertain parameters can be caused by:
parameter insensitivity and/or parameter correlation different parameter values may result in the same good visual fit
Bad (visual) fit fitted data (mass/conc.) does not correspond to measured data
Criteria for an acceptable fit: accuracy of fitted parameter
Criterion 2: accuracy measure: ratio true/fitted
Accurate fitted parameter fitted parameter value is close to the true parameter value.
The ratio was calculated so that the values were smaller or equal to 1
Values close to 1 indicate accurate fit
Possible to fit parameter value that is inaccurate and of which the uncertainty is large but also possible: accurate fitted parameter with large uncertainty
For each fitted value of fNE and kd uncertainty (CV1,PEST conf.intv) and accuracy (true/fitted) is determined
Note:
accuracy of 0.75 is
an arbitrary value:
we decided ourselves that
the fitted value should be
within 25% of the true value
CV1,PEST conf.intv ≤ 0.25 small uncertainty
Criteria for an acceptable fit: uncertainty and accuracy
0
0.2
0.4
0.6
0.8
1
0.01 0.1 1 10 100
Tru
e/f
itte
d
RSE model fit
Small uncertainty &
accurate
Large uncertainty but
accurate
Small uncertainty but inaccurate
Large uncertainty &
inaccurate
CV1,PEST conf.intv
0
0.2
0.4
0.6
0.8
1
0.01 0.1 1 10 100
Tru
e/f
itte
d
RSE model fit
Rightly accepted (small uncertainty
& accurate)
Wrongly rejected (small uncertainty
but accurate)
Wrongly accepted (small uncertainty but
inaccurate)
Rightly rejected (large uncertainty and inaccurate)
A
B
C
D
CV1,PEST conf.intv
Criteria for an acceptable fit: uncertainty and accuracy
Principle:
reliable fitted parameters should always be accurate whatever the degree of uncertainty
Accuracy cannot be used in practice as criterion because the true parameter value is unknown.
Uncertainty can be used as criterion in practice
Criteria for an acceptable fit: uncertainty as criterion
Suppose CV1,PEST conf.intv ≤ 0.25 is criterion for a reliable fiited parameter
How many fits would be rightly accepted and wrongly accepted?
42%
4%
17%
36%
CV 1,PEST conf.intv
AB
C
D
Conclusions
Contour plots of the CV25,true-param of fNE and kd shows that parameter estimation is more successful for larger sorption
The proposed CV1,PEST conf.intv ≤ 0.25 will result in only 4% of the cases in wrongly accepted fits.
Thank you for your attention !
© Wageningen UR
top related