uncertainty quantification and calibration of computer modelshuang251/xian_0212.pdf · uncertainty...

Uncertainty Quantification Emulator Bayesian Calibration Discussion References

Uncertainty quantification and calibration ofcomputer models

Xian He

February 12, 2014

Xian He

Uncertainty quantification and calibration of computer models


Outline

1 Uncertainty QuantificationWhat is uncertainty ?

2 EmulatorWhat and why ?

3 Bayesian CalibrationModel componentsComputer model emulaltorDiscrepancy modelComplete model

4 Discussion

Xian He



Introduction

Physical model is defined by a set of differential equations.

Such mathematical models are implemented as computercodes

To make predictions using the computer codes, somenecessary parameters need to be estimated

Calibration : with observations and computer output, we learnabout the parameters.

Bayesian calibration method allows for all sources ofuncertainty and attempts to correct for the inadequacy of themodel.

Xian He



What is uncertainty ?

Inaccurate inputs

Xian He




Initial/boundary condition uncertainty

From :www.easterbrook.ca

Xian He


www.easterbrook.ca



Inherent randomness of the system

TEM Terrestrial Ecosystem model(TEM)Xiao et al. [1998]

Xian He




Parameters uncertainty

In TEM, the 6 key parameters unknown Chen et al. [2011]

Cmax Maximium rate of photosynthesis C, ranges[50, 1500]gm−2mon−1

KI Half saturation constant for PAR used by plants, ranges[20, 600]Jcm−2d−1

KC Half saturation constant for C02 − C uptake by plants, ranges[20, 600]µLL−1

Tmax Maximum temperature for GPP, range [25, 35]◦C

Tmin Minimum temperature for GPP, range [−12,−1]◦C

Topt Maximum optimum temperature for GPP, range [15, 25]◦C

Xian He




He and Zhang [2012] has reviewed the methods of uncertaintyquantification. Here are three popular approaches

Monte Carlo simulation and variants, Doubilet et al. [1984]

Polynomial Chaos methods, Xiu and Em Karniadakis [2002]

Statistical emulation, OHagan [2006] and Conti and O’Hagan[2010]

Xian He



Outline




4 Discussion

Xian He



What and why ?

What’s an emulator ?

Definition : A surrogate model and a statistical representationof the computer model.

Examples :

Gaussian process models Sacks et al. [1989] and Conti andO’Hagan [2010]Polynomial Chaos expansions

Implementations :

Climate modeling, Rougier et al. [2009]Environmental science, Challenor et al. [2006]Medical science, Dev et al. [2002]

Xian He



What and why ?

What’s an emulator ?

Explanation in flow-chart

Xian He



What and why ?

Why emulator ?

Resembles the behaviour of the real system.

An approximation of complex computer model.

Simpler/easier surrogate of the computer model.

much faster to run than the computer model.

Makes fully probabilistic predictions of what the simulatorwould produce

By giving a probability distribution of the model output, wecould quantify uncertainties.

predicts output at training data points without uncertainty

predict outputs at another point with non-zero variance

Given enough training data, it would predict simulator outputat any desired accuracy

Xian He



What and why ?

Why emulator ?

Prediction

Approximate the behavior of computer models

Given any input x, we could make predictions

Calibration

By comparing observation with emulator prediction, we canestimate the parameters that minimizes the discrepancies.

Xian He



What and why ?

When to use emulator ?

Prediction

The simulator output vary smoothly in response to inputs

The simulator is computer expensive

The dimension of inputs is not high

Xian He



What and why ?

Reference papers

The development of emulator

Sacks et al. [1989] first introduced the emulator for computermodels

Kennedy and O’Hagan [2001] developed the framework ofBayesian calibration of computer models which accounts forall the possible uncertainties

Conti and O’Hagan [2010] extended the emulator tomulti-output

Xian He



Outline




4 Discussion

Xian He



What’s calibration ?

Definition : Model calibration is the process to identifyparameter values θ for which model outputs agrees withobservation data d .

Approaches :

Traditional method which is ad hoc fitting,see Chen andZhuang [2012]Bayesian calibration which provides parameter estimates anduncertainty of parameters,see Kennedy and O’Hagan [2001]

Xian He



Advantages of Bayesian calibration

Allows for all sources of uncertainty

Attempt to correct for any inadequacy of the model, revealedby discrepancy between data and computer model.

Xian He



Model components

The Bayesian calibration framework in Kennedy and O’Hagan[2001] :

z(x) = y(x,θ) + δ(x) + ǫ(x) (1)

θ :the unknown parameters, to be calibrated

x ∈ X : the variables inputs, such as temperature,precipitation.

z(x) : the real observations

y(x,θ) : the emulator of the computer model

δ(x) : discrepancy between z(x) and y(x,θ) besidesmeasurement error ǫ(x)

Measurement error ǫ(x) : observational error which is iidN(0, σ2), where σ2 is estimated by experts.

Xian He



Model components

Model notations

n : number of observations (z)

N : number of computer output ( y)

t1, t2, ..., tN : set of calibration parameters, given by samplingfrom the prior

x∗1, x∗

2, ..., x∗

N : set of variable inputs at which we get thecomputer output y(x∗, t).

x1, x2, ..., xn : set of variable inputs at which we observe theoutput z.

y(x∗i , tk) : set of computer outputs given x∗1, x∗

2, ..., x∗

N andt1, t2, ..., tN

z(xj) : the observations at x1, x2, ..., xn

Xian He



Computer model emulaltor

Here is how to construct an emulator :

Suppose we can afford to obtain output y(si ) at inputx ∈ S = {s1, ..., sn}, n is training sample size.

Next, we establish some statistical relationship betweenoutput function y(x) and the obtained y(si ).

Quantify the uncertainty of y(x) from the statisticalrelationship of y(x) and y(si ).

A Gaussian process emulator

probability distribution for an unknown function

conditional on some unknown parameters θ

estimated from training data

Xian He




Gaussian process emulator : y(x, θ)

The computer emulator with unknown parameters θ is representedby y(x,θ). It is modelled by Gaussian process as follows :

y = η(x,θ) ∼ N(m1(x,θ), c1((x,θ), (x′,θ)))

The mean function m1(x,θ) is models as regression component.For example, hi(x,θ) are features, then

m1(x,θ) = β0 + β1(h1(x,θ)) + · · ·+ βp(hp(x,θ))

where βi are estimated from training set.

Xian He




Gaussian process emulator y(x, θ)

The model of covariance function c1((x,θ), (x′,θ)) is the key step.

It should satisfy

smoothness of y

correction between x and x′

should be close to 1 if x and x′

are sufficiently close

Borrowed from Kennedy and O’Hagan [2001], a proportional modelis chosen when assumed stationary

c1((x,θ), (x′,θ)) = σ2R(x− x

′

)

Xian He



Discrepancy model : δ(x)

δ(x) is normal distribution :

δ ∼ N(m2(x), c2(x, x′))

The measurement error ǫ ∼ N(0, λ0), λ0 is elicited from expert.

m1(x , ) = h1(x, )Tβ

1

m2(x) = h2(x)Tβ2, β = (βT

1 ,βT2 )

T

c1(., .) and c2(., .) have hyperparameters φ1 and φ2

prior information about θ is independent of the others.

Xian He



Posterior Distribution

We would never know the true θ. The purpose of calibration is forfurther using of the computer models.Posterior distributions

D1 = (x∗1, t1), ..., (x∗

N , tN) : set of points where y are available

D2 = (x1,θ), ..., (xn,θ) : set of points where z are observed

H1(D1) : matrix of h1(x∗

1, t1), ..., h1(x∗

N , tN)

the expectation of y is H1(D1)β1

the expectation of z is H1(D2)β1 + H2(D2)β2

V1(D1) is the variance matrix of y, N × N matrix.

V2(D2) is the variance matrix of z, n × n matrix.

Xian He



Posterior distributions

The full set of data dT = (yT , zT ) is normal with the followingmean and covariance

E (d|θ,β,φ) = md (θ) = H(θ)β

var(d|θ,β,φ) = Vd (θ)

Where

H(θ) =

(

H1(D1) 0H1(D2(θ)) H2(D2)

)

and

Vd(θ) =

(

V1(D1) C1D1,D2(θ)T

C1D1,D2(θ)T V1D2(θ) + V2(D2)

)

Xian He



Posterior distributions

With specification of prior, we could get the full posteriordistribution

p(θ,β,φ|d) ∝ p(θ)p(φ)f (d ;md (θ),Vd (θ) (2)

We could integrate β out from (2). Then, we need to estimate thehyperparameters φ from z and y. Finally, we could get theposterior p(θ|d).

Xian He



Outline




4 Discussion

Xian He



The shorting comings of Bayesian calibration in Kennedy andO’Hagan [2001]

Inefficient when the output is multivariate

When the output is time varying time series, the dimension ofy would be very high. Estimation of covariance function wouldbe difficult.

Kennedy and O’Hagan [2001] applies well when n << N.Now, data arrives sequentially that n >> N.

Xian He



How to overcome ?

One way to overcome the time series output is to consider dynamicemulator for the computer model. There are some reference papers

Liu and West [2009] describes a dynamic emulator fordynamic computer model.

Williamson and Blaker [2014] develops Bayesian dynamiclinear model Gaussian processes for emulation of time seriesoutput.

Xian He



Peter G Challenor, Robin KS Hankin, and Robert Marsh. Towardsthe probability of rapid climate change. 2006.

M. Chen, Q. Zhuang, D. R. Cook, R. Coulter, M. Pekour, R. L.Scott, J. W. Munger, and K. Bible. Quantification of terrestrialecosystem carbon dynamics in the conterminous united statescombining a process-based biogeochemical model and MODISand AmeriFlux data. Biogeosciences, 8(9) :2665–2688,September 2011. ISSN 1726-4189.

Min Chen and Qianlai Zhuang. Spatially explicit parameterizationof a terrestrial ecosystem model and its application to thequantification of carbon dynamics of forest ecosystems in theconterminous united states. Earth Interactions, 16(5) :1–22,April 2012. ISSN 1087-3562.

Stefano Conti and Anthony O’Hagan. Bayesian emulation ofcomplex multi-output and dynamic computer models. Journal of

Xian He



Statistical Planning and Inference, 140(3) :640–651, March2010. ISSN 0378-3758.

Parvati Dev, Dale Harris, David Gutierrez, Amol Shah, andS Senger. End-to-end performance measurement of internetbased medical applications. In Proceedings of the AMIASymposium, page 205. American Medical InformaticsAssociation, 2002.

Peter Doubilet, Colin B Begg, Milton C Weinstein, Peter Braun,and Barbara J McNeil. Probabilistic sensitivity analysis usingmonte carlo simulation. a practical approach. Medical decisionmaking : an international journal of the Society for MedicalDecision Making, 5(2) :157–177, 1984.

Xian He and Hao Zhang. Uncertainty quantification, physicalmodels. In Encyclopedia of Environmetrics. John Wiley & Sons,Ltd, 2012.

Xian He



Marc C. Kennedy and Anthony O’Hagan. Bayesian calibration ofcomputer models. Journal of the Royal Statistical Society :Series B (Statistical Methodology), 63(3) :425–464, 2001. ISSN1467-9868.

Fei Liu and Mike West. A dynamic modelling strategy for bayesiancomputer model emulation. Bayesian Analysis, 4(2) :393–411,June 2009. ISSN 1936-0975.

Anthony OHagan. Bayesian analysis of computer code outputs : atutorial. Reliability Engineering & System Safety, 91(10) :1290–1300, 2006.

Jonathan Rougier, David MH Sexton, James M Murphy, andDavid Stainforth. Analyzing the climate sensitivity of thehadsm3 climate model using ensembles from different butrelated experiments. Journal of Climate, 22(13), 2009.

Xian He



Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P.Wynn. Design and analysis of computer experiments. Statist.Sci., 4(4) :409–423, November 1989. ISSN 0883-4237.

Daniel Williamson and Adam Blaker. Evolving bayesian emulatorsfor structured chaotic time series, with application to largeclimate models. SIAM Journal of Uncertainty Quantification, 2(1) :1–28, January 2014.

X. Xiao, J. M. Melillo, D. W. Kicklighter, A. D. McGuire, R. G.Prinn, C. Wang, P. H. Stone, and A. Sokolov. Transient climatechange and net ecosystem production of the terrestrialbiosphere. Global Biogeochemical Cycles, 12(2) :345–360, 1998.ISSN 1944-9224.

Dongbin Xiu and George Em Karniadakis. Modeling uncertainty insteady state diffusion problems via generalized polynomial chaos.

Xian He



Computer Methods in Applied Mechanics and Engineering, 191(43) :4927–4948, 2002.

Xian He


uncertainty quantification and calibration of computer modelshuang251/xian_0212.pdf · uncertainty...

Documents