probabilistic numerical methods for deterministic ... · outline 1 introduction 2 ordinary...

Probabilistic Numerical Methods forDeterministic Differential Equations

Patrick Conrad†, Mark Girolami†, Simo Sarkka∗,Andrew Stuart∗∗ and Konstantinos Zygalakis††

†Department of Statistics, University of Warwick, U.K.∗Aalto BECS, Finland.

∗∗Department of Mathematics, University of Warwick, U.K.†† Mathematical Department, University of Southampton, U.K.

Swiss Numerical Analysis Colloqium,University of Geneva,

April 17th, 2015Funded by EPSRC, ERC, ONR(Warwick University) Probabilistic Numerical Methods Andrew Stuart 1 / 32

Outline

1 Introduction

2 Ordinary Differential Equations

3 Elliptic PDE

4 Conclusions

(Warwick University) Probabilistic Numerical Methods Andrew Stuart 2 / 32

Outline

1 Introduction


3 Elliptic PDE

4 Conclusions


Uncertainty Quantification (UQ)


Bayesian Inverse Problem (BIP)


BIP and UQ


Deterministic Approach to Numerical Approximation of G

Assumption on Numerical Integrator

Approximate forward map G by a numerical method to obtain GN :‖G(u)−GN(u)‖ ≤ ψ(N)→ 0

as N →∞. Leads to approximate posterior measure µN in place of µ.

TheoremFor appropriate class of test functions f : X → S:

‖Eµf (u)− EµNf (u)‖S ≤ Cψ(N).

S.L. Cotter, M. Dashti and A. M . StuartApproximation of Bayesian Inverse Problems.SINUM 2010 48(2010) 322–345.


Probabilistic Approach to Numerical Approximation of G

Approximate G by a random map GN,ω.

Ensure that: E‖G(u)−GN,ω(u)‖ ≤ ψ(N).

Vanilla UQ: infer scale parameter σ in GN,ω(u).BIP UQ: augment unknown input parameters u by ω.

J. SkillingBayesian solution of ordinary differential equationsMaximum Entropy and Bayesian Methods 1992, 23–37.

O.A. Chkrebtii, D.A. Campbell, M.A. Girolami, B. CalderheadBayesian uncertainty quantification for differential equationsarxiv.1306.2365

M. Schober, D.K. Duvenaud and P. HennigProbabilistic ODE solvers with Runge- Kutta meansNIPS 2014, 739–747.

P. Conrad, M. Girolami, S. Sarkka, A.M. Stuart and K. Zygalakisarxiv.1506.04592.


Outline

1 Introduction


3 Elliptic PDE

4 Conclusions


Set-Up

Consider the ODE:dudt

= f (u), u(0) = u0.

One-step numerical methodFor Uk ≈ u(kh):

Uk+1 = Ψh(Uk ), U0 = u0.

Randomized numerical methodFor Uk ≈ u(kh):

Uk+1 = Ψh(Uk ) + ξk (h), U0 = u0,

where ξk (·) is a Gaussian random field defined on [0,h].


Derivation (Euler)

Integral equationFor uk = u(kh) and for t ∈ [tk , tk+1]:

u(t) = uk +

∫ tk+1

tkf(u(s)

)ds

= uk +

∫ t

tkg(s)ds.

Uncertain g

We do not know g(s). Assume that g is a Gaussian random fieldconditioned to satisfy g(tk ) = f (Uk ). This gives approximation U(t) fort ∈ [tk , tk+1]:

U(t) = Uk + (t − tk )f (Uk ) + ξk (t − tk ).

Uk+1 = Uk + hf (Uk ) + ξk (h).


Assumptions

Assumption 1 (Uncertain Approximation)

Let ξ(t) :=∫ t

0 χ(s)ds with χ ∼ N(0,Ch). Then there exists K > 0,p ≥ 1such that, for all t ∈ [0,h], E|ξ(t)ξ(t)T |2F ≤ Kt2p+1; in particularE|ξ(t)|2 ≤ Kt2p+1. Furthermore we assume the existence of matrix Q,independent of h, such that Eξ(h)ξ(h)T = Qh2p+1.

Assumption 2 (Underlying Deterministic Integrator)The function f and a sufficient number of its derivatives are boundeduniformly in Rn in order to ensure that f is globally Lipschitz and thatthe numerical flow-map Ψh has uniform local truncation error of orderq + 1 with respect to the true flow-map Φh:

supu∈Rn

|Ψt (u)− Φt (u)| ≤ Ktq+1.


Theorem

TheoremUnder Assumptions 1 and 2 it follows that there is K > 0 such that

sup0≤kh≤T

E|uk − Uk |2 ≤ Kh2 min{p,q}.

Furthermoresup

0≤t≤TE|(u(t)− U(t)| ≤ Khmin{p,q}.

Scaling of NoiseOptimal scaling of noise is p = q.Then deterministic rate of convergence is unaffected.But maximal noise is added to the system.Fit constant σ in scale matrix Q = σI to an error estimator.


ODE Example

FitzHugh-Nagumo ModelWe illustrate the randomized ODE solvers on the FitzHugh-Nagumomodel two-species (V ,R) non-linear oscillator, with parameters(a,b, c).

Governing Equations

dVdt

= c(

V − V 3

3+ R

), (1)

dRdt

= −1c

(V − a + bR) . (2)

Parameter ValuesFor numerical results, we choose fixed initial conditionsV (0) = −1,R(0) = 1, and parameter values (.2, .2,3).


Derivation

Standard basis

10 510 410 310 210 1100101 =.

Random

Indicator

10 610 510 410 310 210 1100 =.

0 5 10 15 2010 710 610 510 410 310 210 1100 =.

Randomized basis

0.6

0.4

0.2

h=.1

h=.05

h=.02

h=.01

h=.005


Convergence of Random SolutionsDraws from the random solver for fixed σ

4

2

0

2

4= . = . = .

0 5 10 15 204

2

0

2

4= .

0 5 10 15 20

= .

0 5 10 15 20

= . ×


Backward Error Analysis

Modified (Stochastic Differential) Equation

duh

dt= f (uh) + hq

q∑l=0

h`f`(uh) +√

Qh2q dWdt

, uh(0) = u0

TheoremUnder Assumptions 1 and 2, for Φ a C∞ function with all derivativesbounded uniformly on Rn, there is a choice of {f`}q`=0 such that weakerror for true equation si given by∣∣∣Φ(u(T )

)− EΦ

(Uk )

)∣∣∣ ≤ Khq, kh = T .

whilst weak error from the modified equation is given by∣∣∣EΦ(uh(T )

)− EΦ

(Uk )

)∣∣∣ ≤ Kh2q+1, kh = T .


Statistical Inference: find θ = (a,b, c) from noisy observations

Inverse Problem

yj = u(tj) + ηj , y = G(θ) + η.

Deterministic SolverSimply replace u(·) by deterministic approximation to obtain

y = Gh(θ) + η.

Use MCMC to compute P(θ|y).

Randomized SolverReplace u(·) by random approximation to obtain

y = Gh(θ, ξ) + η.

Use MCMC to compute∫P(θ, ξ|y)dξ.


FitzHugh-Nagumo Parameter Posterior (Deterministic Solver)Posterior is over-confident at finite h values

h=.1

h=.05

h=.02

h=.01

h=.005

0.0

0.2

0.4

0.1

0

0.1

5

0.2

0

0.2

52.7

2.8

2.9

3.0

3.1

0.0

0.2

0.4

2.7

2.8

2.9

3.0

3.1


FitzHugh-Nagumo Parameter Posterior (Random Solver)Posterior still contains bias, but posterior width reflects error

h=.1

h=.05

h=.02

h=.01

h=.005

0.0

0.2

0.4

0.1

0

0.1

5

0.2

0

0.2

52.7

2.8

2.9

3.0

3.1

0.0

0.2

0.4

2.7

2.8

2.9

3.0

3.1


Outline

1 Introduction


3 Elliptic PDE

4 Conclusions


Set-Up

Weak Form

u ∈ V : a(u, v) = r(v), ∀v ∈ V.

Galerkin Method

uh ∈ Vh : a(uh, v) = r(v), ∀v ∈ Vh.

ThenVh = span{Φj = Φs

j }Jj=1.

Randomized Galerkin Method

Vh comprises small randomized perturbations of the standard Galerkinmethod:

Vh = span{Φj = Φsj + Φr

j}Jj=1.


Derivation

Standard basis

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0x

0.5

0.0

0.5

1.0

1.5

2.0

u

Randomized basis

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0x

0.5

0.0

0.5

1.0

1.5

2.0

u

Standard hat basis

Randomized hat basis


Assumptions

Assumption 1 (Uncertain Approximation)

The {Φrj}Jj=1 are independent, mean zero, Gaussian random fields,

with the same support as the {Φsj }, and satisfying

Φrj (xk ) = 0 ∀ {j , k},

J∑j=1

E‖Φrj‖2a ≤ Ch2q.

Assumption 2 (Underlying Deterministic FEM)The true solution u is in L∞(D). Furthermore the standarddeterministic interpolant of the true solution, defined by

vs :=J∑

j=1

u(xj)Φsj ,

satisfies ‖u − vs‖a ≤ Chp.


Theorem

TheoremUnder Assumptions 1 and 2 it follows that the random approximationuh satisfies

E‖u − uh‖2a ≤ Ch2 min{p,q}.

Corollary (Aubin-Nitsche Duality)Consider the Poisson equation with Dirichlet boundary conditions anda random perturbation of the piecewise linear FEM approximation, withp = q = 1. Under Assumptions 1 and 2 it follows that the randomapproximation uh satisfies

E‖u − uh‖L2 ≤ Ch2.


PDE Example

Standard elliptic inversion problem:

−∇ · (κ(x)∇u(x)) = −4x

u(0) = 0,u(1) = 2

0.0 0.2 0.4 0.6 0.8 1.0x

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

log

0.0 0.2 0.4 0.6 0.8 1.0x

0.5

0.0

0.5

1.0

1.5

2.0

2.5

u

κ(x) =N∑

n=1

θiIi(x).


Statistical Inference: find θ from noisy observations

Inverse Problem

yj = u(tj) + ηj , y = G(θ) + η.

Deterministic SolverSimply replace u(·) by deterministic approximation to obtain

y = Gh(θ) + η.

Use MCMC to compute P(θ|y).

Randomized SolverReplace u(·) by random approximation to obtain

y = Gh(θ, ξ) + η.

Use MCMC to compute∫P(θ, ξ|y)dξ.


Elliptic Inference (Deterministic Solver)

h=1/10

h=1/20

h=1/40

h=1/60

h=1/80

64202

2

0

2

4

3 2 1 0 1

2

0

2

6 4 2 0 2 2 0 2 4 2 0 2


Elliptic Inference (Random Solver)

h=1/10

h=1/20

h=1/40

h=1/60

h=1/80

64202

2

0

2

4

3 2 1 0 1

2

0

2

6 4 2 0 2 2 0 2 4 2 0 2


Outline

1 Introduction


3 Elliptic PDE

4 Conclusions


Summary

Numerical methods are inherently uncertain.

Classical numerical anaysis upper bounds this uncertainty.

Our approach treats it as a random variable.

Mean square rates of convergence are derived, consistent with classicalnumerical analysis.

Backward error analysis gives universal interpretation of the methods assolving stochastic or random problems.

In forward modelling scale parameter is set using classical errorindicators.

In inverse modelling, the random parameters augment the unknowninputs.


References

P. DiaconisBayesian numerical analysis.Statistical Decision Theory and Related Topics IV 1(1988) 163–175.

M. Dashti and A.M. StuartThe Bayesian approach to inverse problems.Handbook of Uncertainty QuantificationEditors: R. Ghanem, D.Higdon and H. Owhadi, Springer, 2016.arXiv:1302.6989

S.L. Cotter, M. Dashti and A. M . StuartApproximation of Bayesian Inverse Problems.SINUM 2010 48(2010) 322–345.

P. Conrad, M. Girolami, S. Sarkka, A.M. Stuart and K. Zygalakisarxiv.1506.04592.


probabilistic numerical methods for deterministic ... · outline 1 introduction 2 ordinary...

Documents