sensitivity and importance analysis

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Sensitivity and Importance Analysis

Charles [email protected]



Sensitivity Analysis Defined

• Study of how the variation in the output of a model can be apportioned, qualitatively or quantitatively, to different “sources of variation” in the inputs for the purpose of increasing confidence in the analysis– Include assumptions– Input uncertainty– Scenario/model uncertainty



The Point

• Complex analysis may have dozens of input and output variables that are linked by a system of equations

• Analysts and decision makers must understand the relative importance of the components of an analysis

• Some outcomes and decisions are sensitive to minor changes in assumptions and input values



Sensitivity Analysis

• If it is not obvious which assumptions and uncertainties most affect outputs, conclusions and decisions the purpose of sensitivity analysis is to systematically find this out



Systematic Investigation of…

• Future scenarios

• Model parameters

• Model inputs

• Assumptions

• Model functional form



Assumptions Sensitivity

• List the key assumptions (scenarios) of your analysis

• Explore what happens as you change/drop each one individually– Do your answers change?

• Challenging assumptions can be effective sensitivity analysis



Input Sensitivity

• Parameter-how sensitive is our output to forecast error or other changes in inputs? Unexpected change or error

• Decision variables (Inputs we control)-might changes in our decisions/actions improve our outputs



Sensitivity Analysis Methods

• Deterministic one-at-a-time analysis of each factor

• Deterministic joint analysis• Scenario analysis• Subjective estimates• Parametric analysis--range of values• Probabilistic analysis can be used for

importance analysis



One-At-A-Time Analysis

• Hold each parameter constant– Expected value– Representative value

• Let one input vary– Assumption– Input– Parameter

• Common, useful, dangerous




• Do not equate magnitude with influence• A=U(107,108), B=U(2,6)• C = A + B; A dominates• C = AB; B dominates




• Dependence and branching in model creates flaws with this logic

If A<50 then

C = B + 1

Else

C = B100

What value do we set A equal to?



Joint Analysis

• Change combinations of variables at same time

• Enables analysts to take dependencies explicitly into account

• Can have same limitations as OAAT analysis



Subjective Estimates

• Subjective estimates of uncertain values can be used to identify threshold values of importance to the risk assessment



Range of Values

• A specific (not subjective) range of values is used– E.g., 10th, 50th, 90th percentiles

• Ceteris paribus approach• All possible combinations approach

– All 10th percentiles, 10th with 90th and so on



Importance Analysis

• How much does each model input contribute to the variation in the output?

• Typically a few key inputs account for most output variation– These are your important inputs.

• Not particularly good at identifying nonlinear or multivariate relationships



Habitat Units CreatedP

rob

ab

ility

HUs

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

30 60 90 120



Regression Sensitivity for Grand Total HUs/D14

Std b Coefficients

B=>15 Degrees C / Use this.../L10 .071

V5: water temperature / Me.../H9 .073

V6: Dissolved oxygen / Mea.../N22-.075

V15: Pool class / Use this.../O26-.082

V6: Dissolved oxygen / Mea.../H22-.105

B=>15 Degrees C / Use this.../O10-.106

V15: Pool class / Use this.../H26-.117

V5: water temperature / Me.../N9 .147

V6: Dissolved oxygen / Mea.../B22-.157

V5: water temperature / Me.../E9 .16

V5: water temperature / Me.../K9 .162

A=resident rainbow trout /.../X6-.163

B=>15 Degrees C / Use this.../U10-.194

B=>15 Degrees C / Use this.../E10 .22

A=resident rainbow trout /.../L6-.287

A=resident rainbow trout /.../R6-.45

-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1



Advanced Statistical Methods

• Apportion variation in output to inputs via– Regression analysis– Analysis of variance– Response surface methods– Fourier amplitude sensitivity test (FAST)– Mutual information index (MII)– Classification and regression trees (CART)



So What?

• When decision is sensitive to changes or uncertainties within realm of possibility then more precision and additional information may be required– More data (research)– Better models– Conservative risk management



Take Away Points

• “What if” analysis is essential to good risk assessment

• Systematic investigations of model parameters, model inputs, assumptions, model functional form

• Essential to good risk management



Charles Yoe, [email protected]

Questions?

sensitivity and importance analysis

Documents