Using the Model to Evaluate Observation Locations and

Parameter Information in the Context of Predictions

Data Worth and PredictionsPredictions:• Most models are developed with the intention of making predictions

about future system behavior• Predictions can also be posed as some unknown occurrence at some

location in the model domain that can be simulated by the model – such as flow in an unmonitored stream reach

Approach :• The best possible model is developed and calibrated under the

assumption that a model that reproduces past system response and system properties makes reliable predictions

• But does the best fitting model always produces the best predictions?

Evaluations (questions we should ask!):• How much uncertainty accompanies simulated predictions?• Which observations and parameters most influence the value of this

predictive uncertainty?

Predictions – Death Valley

Model:• Hydrogeology described

by many parameters• Large number of head

and flow observations to calibrate the model

Prediction:• Groundwater flow

directions and velocities in the Yucca Flats area

Predictions – Death Valley

Prediction• Advection is often the

dominant aspect of groundwater transport

• Advection can be simulated using particle tracking or path lines

• This is available within MODFLOW using the Advection (ADV) package

• MODFLOW can also provide sensitivities pertaining to the pathway

Predictions – AdvectionPrediction• Specifically – MODFLOW

calculates the travel path in three directions:• X - East-West• Y - North-South• Z - Up-Down

• Using calculations described later, the variance of these predictions can be easily determined

Parameters – Predictions (PPR)

1. Parameter Information

• Which parameters are most important to predictions

• What information might be cost-effectively collected to further reduce predictive uncertainty?

Observations – Predictions (OPR)

2. Observation Locations

• Existing observationsExisting observations: Which existing observation locations are most influential?

• New observationsNew observations:: Where could new measurements be made to reduce predictive uncertainty?

Data Worth and PredictionsEvaluations:• How much uncertainty accompanies simulated

predictions?• Evaluate deterministically – trial and error sensitivity.• Evaluate using multiple models and Monte-Carlo methods• Evaluate using statistics – variance, standard deviation

• Which parameters and observations are most influential in the calculation of this predictive uncertainty?• Statistical methods enable rapid evaluation of the

contribution of (a) parameters and / or (b) observations to the uncertainty (standard deviation) of one or more predictions

• First order second moment (FOSM) methods• First order – linear sensitivities• Second moment – variances and standard deviations

Predictions – Uncertainty

Standard Deviation• Measure of spread of values

for a variable• Has a stochastic basis• Involves assumptions• Regardless – a means for

comparing relative predictive uncertainty

Normal distribution

Data Worth and PredictionsApproach:• Define the prediction:

• How can we simulate the predicted quantity?• How can we summarize the ‘value’ of the prediction

numerically?

• Gather information obtained through calibration:• Observation sensitivities and weights• Prior information on parameters

• Calculate prediction sensitivities:• Using methods similar to when calculating observation

sensitivities• Include all parameters if possible

• calibrated parameter values • measured parameter values

Data Worth and PredictionsApproach:• Calculate the current prediction variance• Calculate a hypothetical prediction variance assuming changes

to (a) information about parameters or (b) available observations

• The Parameter-PRediction (PPR) Statistic:• Evaluate worth of potential knowledge about parameters, posed in

the form of prior information - add this to calculations

• The Observation-PRediction (OPR) Statistic:• Evaluate existing observation locations by omitting them from the

calculations • Evaluate potential new observation locations by adding them to the

calculations

Note: PPR was called Value of Improved Information (VOII) in one journal article

[1- (sznew / sz)] x 100

Data Worth and Predictions

Determine Predictive Uncertainty

EVALUATE CURRENT CONDITIONS

EVALUATE POTENTIAL

OBSERVATIONS

EVALUATE EXISTING

OBSERVATIONS

CALCULATE & REPORT PREDICTION

STATISTICS

EVALUATE POTENTIAL

PARAMETER DATA

Determine Predictive Uncertainty

EVALUATE CURRENT CONDITIONS

EVALUATE POTENTIAL

OBSERVATIONS

EVALUATE EXISTING

OBSERVATIONS

CALCULATE & REPORT PREDICTION

STATISTICS

EVALUATE POTENTIAL

PARAMETER DATA

Data Worth and Predictions

Approach:• Recently encapsulated in a program:

• OPR-PPR Program• In review

• Class Exercise will use the OPR and PPR methods with a synthetic model that is described both in the OPR-PPR documentation, and in Hill and Tiedeman (2007)

Relevant ReferencesTiedeman, C.R., Hill, M.C., D’Agnese, F.A., and Faunt, C.C., 2003, Methods

for using groundwater model predictions to guide hydrogeologic data collection, with application to the Death Valley regional ground-water flow system: Water Resources Research, 39(1): 5-1 to 5-17, 10.1029/2001WR001255. (The PPR statistic is the same as what is called the VOII statistic in this paper)

Tiedeman, C.R., D.M. Ely, M.C. Hill, and G.M. O'Brien, 2004, A method for evaluating the importance of system state observations to model predictions, with application to the Death Valley regional groundwater flow system, Water Resources Res., 40, W12411, doi:10.1029/2004WR003313.

Tonkin, M.J., Tiedeman C.R., Ely, D.M., and Hill M.C., (in press), Documentation of OPR-PPR, a Computer Program for Assessing Data Importance to Model Predictions Using Linear Statistics, Constructed using the JUPITER API, JUPITER: Joint Universal Parameter IdenTification and Evaluation of Reliability API: Application Programming Interface, U.S. Geological Survey Techniques and Methods Report TM6-E2.