sensitivity and importance analysis

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Sensitivity and Importance Analysis

Charles Yoecyoe1@verzion.net

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Sensitivity Analysis Defined

• Study of how the variation in the output of a model can be apportioned, qualitatively or quantitatively, to different “sources of variation” in the inputs for the purpose of increasing confidence in the analysis– Include assumptions– Input uncertainty– Scenario/model uncertainty

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

The Point

• Complex analysis may have dozens of input and output variables that are linked by a system of equations

• Analysts and decision makers must understand the relative importance of the components of an analysis

• Some outcomes and decisions are sensitive to minor changes in assumptions and input values

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Sensitivity Analysis

• If it is not obvious which assumptions and uncertainties most affect outputs, conclusions and decisions the purpose of sensitivity analysis is to systematically find this out

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Systematic Investigation of…

• Future scenarios

• Model parameters

• Model inputs

• Assumptions

• Model functional form

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Assumptions Sensitivity

• List the key assumptions (scenarios) of your analysis

• Explore what happens as you change/drop each one individually– Do your answers change?

• Challenging assumptions can be effective sensitivity analysis

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Input Sensitivity

• Parameter-how sensitive is our output to forecast error or other changes in inputs? Unexpected change or error

• Decision variables (Inputs we control)-might changes in our decisions/actions improve our outputs

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Sensitivity Analysis Methods

• Deterministic one-at-a-time analysis of each factor

• Deterministic joint analysis• Scenario analysis• Subjective estimates• Parametric analysis--range of values• Probabilistic analysis can be used for

importance analysis

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

One-At-A-Time Analysis

• Hold each parameter constant– Expected value– Representative value

• Let one input vary– Assumption– Input– Parameter

• Common, useful, dangerous

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

One-At-A-Time Analysis

• Do not equate magnitude with influence• A=U(107,108), B=U(2,6)• C = A + B; A dominates• C = AB; B dominates

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

One-At-A-Time Analysis

• Dependence and branching in model creates flaws with this logic

If A<50 then

C = B + 1

Else

C = B100

What value do we set A equal to?

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Joint Analysis

• Change combinations of variables at same time

• Enables analysts to take dependencies explicitly into account

• Can have same limitations as OAAT analysis

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Subjective Estimates

• Subjective estimates of uncertain values can be used to identify threshold values of importance to the risk assessment

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Range of Values

• A specific (not subjective) range of values is used– E.g., 10th, 50th, 90th percentiles

• Ceteris paribus approach• All possible combinations approach

– All 10th percentiles, 10th with 90th and so on

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Importance Analysis

• How much does each model input contribute to the variation in the output?

• Typically a few key inputs account for most output variation– These are your important inputs.

• Not particularly good at identifying nonlinear or multivariate relationships

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Habitat Units CreatedP

rob

ab

ility

HUs

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

30 60 90 120

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Regression Sensitivity for Grand Total HUs/D14

Std b Coefficients

B=>15 Degrees C / Use this.../L10 .071

V5: water temperature / Me.../H9 .073

V6: Dissolved oxygen / Mea.../N22-.075

V15: Pool class / Use this.../O26-.082

V6: Dissolved oxygen / Mea.../H22-.105

B=>15 Degrees C / Use this.../O10-.106

V15: Pool class / Use this.../H26-.117

V5: water temperature / Me.../N9 .147

V6: Dissolved oxygen / Mea.../B22-.157

V5: water temperature / Me.../E9 .16

V5: water temperature / Me.../K9 .162

A=resident rainbow trout /.../X6-.163

B=>15 Degrees C / Use this.../U10-.194

B=>15 Degrees C / Use this.../E10 .22

A=resident rainbow trout /.../L6-.287

A=resident rainbow trout /.../R6-.45

-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Advanced Statistical Methods

• Apportion variation in output to inputs via– Regression analysis– Analysis of variance– Response surface methods– Fourier amplitude sensitivity test (FAST)– Mutual information index (MII)– Classification and regression trees (CART)

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

So What?

• When decision is sensitive to changes or uncertainties within realm of possibility then more precision and additional information may be required– More data (research)– Better models– Conservative risk management

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Take Away Points

• “What if” analysis is essential to good risk assessment

• Systematic investigations of model parameters, model inputs, assumptions, model functional form

• Essential to good risk management

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Charles Yoe, Ph.D.cyoe1@verizon.net

Questions?