polsko-norweski fundusz badań naukowych / polish-norwegian research fund estimation of uncertainty...

Polsko-Norweski Fundusz Badań Naukowych / Polish-Norwegian Research Fund Polsko-Norweski Fundusz Badań Naukowych / Polish-Norwegian Research Fund

Estimation of uncertainty in status class assessment

for Wel waterbodies

Jannicke Moe (NIVA)

deWELopment project meeting

10.02.2011, Warszaw

Polsko-Norweski Fundusz Badań Naukowych / Polish-Norwegian Research Fund

From the deWELopment project description:

Third phase: from single metrics to BQE-level assessment

• We will test different methods of combination of these single metric results to obtain a total result at the whole element level, taking into account the uncertainty in the different single metrics

• In our project we will test alternative approaches [ to the one-out-all-out principle] using different methods– simple averaging, weighted averaging, multimetric approach


From the deWELopment project description:

Fourth phase: from BQE-level to waterbody-level assessment

• Testing different ways of combining the assessment results for different BQEs into one final result for the whole waterbody. – Here the recommended by one-out-all-out rule will be compared

to other alternative methods.

• The risk of misclassification will be estimated– software STARBUGS ( WISERBUGS)


Outline

• Uncertainty and risk of misclassification at BQE level

• Integration of uncertainty from BQE level to waterbody level

• WISERBUGS tool: examples with deWELopment results

• Next steps


Uncertainty and risk of misclassification at BQE level


Uncertainty: "accuracy" vs. "precision"

• High accuracy, but low precision– "roughly right"

• High accuracy and high precision– optimal result

• Low accuracy, but high precision– "precisely wrong"

metric

True value

metric

True valuemetric

True value

5

5

5

• We can never know the ”true value”of a BQE metric – only the measured value

• Standard Deviation (SD) is a measure of precision, not accuracy


Uncertainty in BQE level

• If we can never know the ”true value” of a BQE metric – how can we say something about uncertainty???

• We must assume that the measured mean value represents the true value and the true status class

• We can assume that measured metric values follow normal distribution due to sampling uncertainty

• We can let measured SD represent sampling uncertainty

Example:Measured metric values: 3, 5, 5, 6, 6Mean: 5Standard Deviation: 1.22

Histogram of DATA

De

nsi

ty

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

BQE metric value0 2 4 6 8 10


Risk of misclassification - BQE level

• With the given mean and SD, we can test: if we re-sample the BQE many times, how often will we get the ”true” status class?

• Assuming that measured status class = true status class

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

BQE metric value

pro

ba

bili

ty d

en

sity

fun

ctio

n

meanSD = 1.22

p=0.7% p=20% p=58.6% p=20% p=0.7% Risk of misclassification:

= proportion of ”new samples” which result in wrong status class

= 0.7% + 20% + 20% + 0.7%

= 41.4%


0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

BQE value

pro

ba

bili

ty d

en

sity

fun

ctio

nmean

SD = 1.5

p=2.3% p=23% p=49.5% p=23% p=2.2%

Risk of misclassifcation increases with SD

- Higher SD gives flatter distribution=> higher probability that new BQE values fall outside the true class

Example:

- SD increases from 1.0 to 1.5

- Probability of misclassification increases from 32% to 50%

- ”True class” is Moderate, but 25% probability that new samples will result in Good or High

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

BQE value

pro

ba

bili

ty d

en

sity

fun

ctio

n

meanSD = 1

p=0.1% p=15.7% p=68.3% p=15.7% p=0.1%


Risk of miscl. increases near class borders

- If the measured mean BQE is close to a class border, then new BQE values are more likely to fall into a neighbour class

Example:

- Mean BQE decreases from 5 to 4.2

- Probability of misclassification increases from 32% to 46%

- ”True class” is moderate, but 42% probability that new samples will result in Good or High

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

BQE value

pro

ba

bili

ty d

en

sity

fun

ctio

n

meanSD = 1

p=0.1% p=15.7% p=68.3% p=15.7% p=0.1%

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

BQE value

pro

ba

bili

ty d

en

sity

fun

ctio

nmean

SD = 1

p=1.4% p=40.7% p=54.3% p=3.6% p=0%


How can we obtain SD for BQE level?

• Can we get reliable SD estimates from deWELopment data?– Many samples are needed per BQE and waterbody –

we have ”only” 1-2 samples per BQE and waterbody (?)– Metric values are not necessarily normally distributed

• Other distributions can be considered

• What are the alternatives?– ”Use best-available information from replicated sampling studies

on environmentally similar waterbodies” (WISER data?)– Use best guesses– (Ask WISER WP6.1 for advice)


Integration of uncertainty from BQE level to waterbody level


Macroinvertebrates

Phytobenthos

Hydrology

Acidification

Organic

Combining metrics and BQEs


Two issues:1) How to combine status classes for different metrics and BQEs

• Average; weighted average; all-out-one-out; etc. • Will not be discussed here (see my presentation June 2010)

2) How to combine uncertainty from different metrics and BQEs • Tool: WISERBUGS

Combining metrics and BQEs


WISERBUGS – brief introduction

• WISER Bioassessment Uncertainty Guidance Software

• Excel-based tool developed within EU project WISER

• Purpose: assist in quantifying uncertainty in the assessment of ecological status of waterbodies

• Can be used for testing impact on classification of:– Combination rules for metrics and for BQEs– Class boundaries and reference conditions– Sampling uncertainty – SD (per metric)– Sorting/identification uncertainty – SD (per metric)– Uncertainty in reference condition – SD (per metric)

• Can not be used for – estimating SD for metrics (must be done separately)– estimating type I/II errors (because true status class is not known)



WISERBUGS output: probability of each status class for...

• each waterbody (overall assessment)

• each BQE within a waterbody

• each metric within a BQE

High Good Moderate Poor Bad

0.0

0.2

0.4

0.6

0.8

Status class

pro

ba

bili

ty



WISERBUGS input

• For each metric:– Measured value for each sample in each waterbody– SD representing sampling variation– Class boundaries (H/G, G/M, M/P, P/B)– ”E1”: metric value for which EQR = 1 (Reference value)– ”E0”: metric value for which EQR = 0 (bottom of metric scale)– (SD representing sorting/identification variation)– (SD for reference value)

• For overall assessment:– Combination rules for metrics within BQE– Combination rules for BQEs within waterbody– (Correlation between metrics)


WISERBUGS tool: examples with deWELopment results


Data received from deWELopment

• Selected metrics:– Rivers:

• MP: River Macrophyte Index

• PB: Multimetric Diatom Index for rivers

• MI: Benthic Macroinvertebrate Index

• FI: European Fish Index +

– Lakes: • PP: Chlorophyll a; Phytoplankton Metric for Polish Lakes

• MP: Ecological State Macrophyte Index

• PB: Diatom Index for Lakes

• MI: Benthic Quality Index based on Chironomid Pupal Exuvial Techn.

• FI: Fish Index 'Summ Best’ (no class boundaries yet)


Data received from deWELopment

• Metric values:– 13 rivers + 10 lakes– Usually all BQEs for each waterbody

• Standard deviations per metric: not available– Randomised values used for this excercise– (To be discussed)

• Class boundaries and reference conditions (”E1”) per metric:– Sometimes waterbody-specific (OK for WISERBUGS)

• ”E0” – given in correct scale?


Metric specification - 1 (Lakes)

Class boundaries

Metric names

Other details



Other types of variation

SD from sampling variation

NB: SD values are made up!



Grouping of metrics by pressure within BQE

Grouping of metrics by BQE



Weighting of each BQE

Rule for combining BQEs within waterbody(here: one-out-all-out)

Rule for combining metrics within BQE


Results: assessment for individual metrics

NB: Fake SD values - results must not be interpreted as real.


Results: assessment combined per BQE and per waterbody

• Combination rule for total assessment: ”worst case” (all 4 BQEs)


Results: assessment for BQEs and waterbody

• Combination rule for total assessment: ”average” (all 4 BQEs)


Next steps


Next steps for data analysis

• Try to obtain SD estimates / best guesses from BQE groups

• Quality-check of class boundaries, E0, E1 etc. – Confusion EQR scale vs. original metric scale?

• Explore impact on classification of...– different level of uncertainty (sampling SD)– different combination rules– etc.– what is most useful for deWELopment?

• Estimate risk of misclassification for selected cases– ”True class” will be determined by the given metric values,

and an agreed set of SD and combination rules

• Include physico-chemical parameters in assessment?

• Other suggestions?


Publication

• 1 manuscript on exploring risk of misclassification using the metrics and classification system developed by deWELopment and the WISERBUGS tool

• Potential co-authors– BQE group leaders ? – Gosia ? – Coordinators: Hanna and Anne (or acknowledgement?)– WISERBUGS author: Ralph Clarke (or acknowledgement?)

polsko-norweski fundusz badań naukowych / polish-norwegian research fund estimation of uncertainty...

Documents

bqe level slide

bqe metric

bqelevel assessment

accuracy slide

measured status class

element level

status class assessment

measured mean value