physics-informed recurrent neural networks for...

Physics-Informed Recurrent Neural Networksfor Land Model Surrogate Construction

Vishagan Ratnaswamy 1 Cosmin Safta 1 Khachik Sargsyan 1

Daniel M. Ricciuto 2

1 Sandia National Laboratories, Livermore CA

2Oak Ridge National Laboratory, Oak Ridge TN

Vish. Ratnaswamy (Sandia National Labs) Tree-LSTM NN Surrogate Models 1 / 28

Acknowledgements

DOE, Office of Science,

I Advanced Scientific Computing Research (ASCR)I Scientific Discovery through Advanced Computing (SciDAC) programI Biological and Environmental Research (BER)I National Energy Research Scientific Computing Center (NERSC)

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and EngineeringSolutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s

National Nuclear Security Administration under contract DE-NA-0003525.


Outline

MotivationI Energy Exascale Earth System Model (E3SM): Land component

Need for Surrogate ModelsI Polynomial Chaos SurrogateI Neural Network Surrogates

F Multilayer PerceptronF LSTM RNNF Tree-LSTM RNN

Global Sensitivity Analysis (GSA)

Preliminary Results


E3SM Model Overview

Energy Exascale Earth System Model (E3SM) is a coupled earth model

US Department of Energy (DOE) sponsored Earth system model https://e3sm.org

Ocean, Atmosphere, Sea ice and Land Components

Computationally expensive to run the coupled mode (including ocean and atmosphere)

Can only do a few global simulations of coupled E3SM models → hard to get training data


https://e3sm.org

E3SM Land Model (ELM) Overview

Land model incorporates a set of biogeophysical processes:

Cheapest component, can run in single column modeSimplified python model available (sELM)Can evaluate model many times for various input parameters


ELM Produces Time Series givenInput Parameters and Forcing Drivers

Daily Forcings (site dependent)1 Min/Max Temperature2 Day of Year3 Solar radiation4 Water Availability

Thirty year record (1980-2009)of forcings

47D Stochastic Space

QOIs dependent on previousday’s output

initializemodel

Inputs Drivers

20.00 20.25 20.50 20.75 21.00 21.25 21.50 21.75 22.00year since 1980

0

50

100

150

200

250

300

350

FSDS

US-Ha1US-Ha1+0.5 latUS-Ha1-0.5 lat

−2 −1 0 1 2 30

0.2

0.4

0.6

0.8

1

QOI

160 180 200 220 240 260days

5

0

5

10

15

20

GPP

dataMLPLSTMTree-LSTM


Surrogates are necessary for expensive computationalmodels

... otherwise called supervised ML

Surrogates are required for ensemble-intensive studies, such asI parameter estimationI uncertainty propagationI global sensitivity analysisI optimal experimental design

This talk

Investigating surrogate construction approaches for ELMto enable all of the above


Polynomial chaos (PC) surrogate for black-box f (λ)

Represent QOIs as orthogonal expansion of random variables

f (λ) ≈ fc(λ(ξ)) =∑α∈I

cαΨα(ξ)

Germ: ξ = (ξ1, ξ2, ..., ξd), e.g. uniform: λ(ξ) is a linear scaling

Multi-index α = {α1, . . . , αd}Orthogonal polynomials wrt p(ξ), Ψα(ξ) =

∏di=1 ψαi (ξi )

Typical construction approach: regression to find PC modes cαAdvantages of PC

I moment estimation, uncertainty propagation, global sensitivity analysisExpensive model challenge

I Use Bayesian regression, helps to quantify lack of simulation dataHigh-d challenge d � 1

I Number of terms in expansion of order p and dimension d : |I| = (d+p)!d!p!

I Use sparse regression

We employ Bayesian compressed sensing (BCS): iterative algorithmfor Bayesian sparse learning [Babacan, 2010; Sargsyan, 2014; Ricciuto, 2018]


‘Free’ Global Sensitivity Analysis with PC f (λ(ξ)) '∑α∈I

cαΨα(ξ)

Main effect sensitivity indices

Si =Var [E(f (λ|λi )]

Var [f (λ)]=

∑α∈Ii

c2α||Ψα||2∑α∈I\{0}

c2α||Ψα||2

I Ii is the set of bases with only ξi involvedI Si is the uncertainty contribution that is due to i-th parameter only

Total effect sensitivity indices

Ti = 1− Var [E(f (λ|λ−i )]

Var [f (λ)]=

∑α∈ITi

c2α||Ψα||2∑α∈I\{0}

c2α||Ψα||2

I ITi is the set of bases with ξi involved, including all its interactions.I Ti is the total uncertainty contribution due to i-th parameter

[Sudret, 2008; Crestaux, 2009; Sargsyan, 2017]


Surrogate Modeling: MLP Approach

Benign feed-forward networkDoes not account for temporal aspect of model

I Cannot propagate information of QOIs day to day (or month to month)

Model dealt with as a black box

……

……

x1

x2

x3

x47

……

……

……

……

……

……

………………..

Hidden Layer 1 Hidden Layer 2 Hidden Layer N-1

Daily Forcing

Stochastic Input

GPPNEELAINPP


Surrogate Modeling: LSTM-RNN

Vanilla LSTM Recurrent NN architecture

Accounts for temporal aspect of model

Model dealt with as a black box

Day 1 Day 2 Day 3

…………

Day N

x1 x2 x3

…………

x47

Daily Forcing

Stochastic Input

GPPNEELAINPP


Graph Structure of ELM Land ModelLooking under the hood helps build physics-informed architecture

ACMf(TM , Tm,

BTRAN,FSDS)

AutotrophicRespirationf(TM , Tm)

AllocationPhenologyf(TM , Tm)

LitterProcessesf(TM , Tm)

SOMDecomposition

f(TM , Tm)

nue (grass,tree)slatop (everg, decid)

fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd

gdd critcrit daylndays onndays offleaf long

froot longlwtop annfstor2tran

r mortk l(1,2,3)

k fragflig(cwd,fr,lf)

flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)

rf(l1s1,l2s2,l3s3)rf(s1s2,s2s3,s3s4)

soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE


Surrogate Modeling: Tree LSTM RNN

Retain similar structure to Gated RNN ([Tai, 2015])

Incorporate physical connections into LSTM

Each QOI has a LSTM gate

GPP

LAI

QOIDay 1GPP

LAI

QOIDay 2

…………x1 x2 x3 x47

NEE

NPP

NEE

NPP

LSTM RNNVanilla

GPP

LAI

QOIDay 1GPP

LAI

QOIDay 2

…………x1 x2 x3 x47

NEE

NPP

NEE

NPP

Tree-LSTM RNNPhysics-informed


For NN surrogates, GSA can be carried out withMonte-Carlo

Mean estimate: E [f (λ)] ≈ 1N

N∑n=1

f (λ(n))

Variance estimate: Var [f (λ)] ≈ 1N

N∑n=1

f (λ(n))2 − E [f (λ)]2

Main sensitivity:

Si =Var [E(f (λ|λi )]

Var [f (λ)]≈ 1

Var [f (λ)]

(1

N

N∑n=1

f (λ(n))f (λ′(n)−i ∪ λ

(n)i )− E [f (λ)]2

),

where λ′(n)−i ∪ λ

(n)i is a single-column swap sample given two sampling

schemes λ(n) and λ′(n)

... similar estimators for total sensitivity

Inherits all the challenges of Monte-Carlo

[Jansen, 1999; Sobol, 2001; Saltelli, 2002]


Overview of Surrogate Models Implemented

Surrogatemodels

ML

LSTM-RNN

GSA

Tree-LSTMRNN

PCE

GSA

• Di�cult withtime depend.

• Easy tocompute

• Need MC

• No timedepend.

• No hierarchy

• Hierarchy

MLP

Surrogates also needed for inverse UQ (calibration)A


Summary of Case Study

Training Details

Generate samples from sELM model: 30 years (1980-2009)

Each training set (time history) has 10, 950 data points (daily)

Simulation at University of Michigan Biological Station site

500 training samples, 500 validation samples

NN Details

Train on daily QoIs

500 Epochs of SGD

2 layers, 150 neurons

L2 loss, dropout regularization

MLP

No time dynamics

No physics

LSTM

Time dynamics

No physics

Tree-LSTM

Time dynamics

Physics

PC Details

Hard to train on dailyaverages (noisy)

Train on monthly averages

Use Bayesian compressivesensing to computecoefficients

Build surrogate for eachaverage month, i.e. 30× 12surrogates


Snapshot of training sample 1

160 180 200 220 240 260day

2

4

6

8

10

12

14

16

GPP

MLPLSTMTree-LSTMdata


Snapshot of GPP training sample 2

160 180 200 220 240 260day

2

4

6

8

10

12

14

16

GPP



Snapshot of GPP training sample 3

160 180 200 220 240 260day

2

4

6

8

10

12

14

16

GPP



Snapshot of GPP validation sample 1

160 180 200 220 240 260day

2

4

6

8

10

12

14

16

GPP




160 180 200 220 240 260day

2

4

6

8

10

12

14

16

GPP



GPP Parity Plot comparisons for various months

5 0 5 10 15sELM

5

0

5

10

15Su

rroga

te

March

MLPLSTMPCETree-LSTM



5 0 5 10 15sELM

5

0

5

10

15Su

rroga

te

August

MLPLSTMPCETree-LSTM



5 0 5 10 15sELM

5

0

5

10

15Su

rroga

te

December

MLPLSTMPCETree-LSTM


Global Sensitivity Analysis Comparison

gdd_

crit

crit_

dayl

nday

s_on

nday

s_of

fnu

e_tre

enu

e_gr

ass

slato

p_ev

erg

slato

p_de

cidliv

ewdc

nle

afcn

froot

cnfs

tor2

tran

stem

_leaf

croo

t_st

emf_

livew

dfro

ot_le

afrg

_fra

cbr

_mr

q10_

mr

csto

r_ta

ur_

mor

tlw

top_

ann

leaf

_long

froot

_long

q10_

hrk_

l1k_

l2k_

l3k_

s1k_

s2k_

s3k_

s4k_

frag

rf_l1

s1rf_

l2s2

rf_l3

s3rf_

s1s2

rf_s2

s3rf_

s3s4

soil4

cicw

d_fli

gfr_

flig

lf_fli

gfr_

flab

lf_fla

b fpi

fpg

Parameter

0.00

0.05

0.10

0.15

0.20

0.25

S iGSA comparison for PCE and Tree-LSTM RNN

PCEMLPLSTM RNNTree LSTM RNN


Tree RNN more accurate than PCE and traditional MLmethods

Computed Mean RMS for each Surrogate

Method Train (Daily/Month) Val (Daily/Month)

PCE (N/A)/35% (N/A)/46%MLP 19/14% 32/20%

LSTM 14/10% 21/16%Tree-LSTM 6/2% 9/5%

Tree-LSTM outperforms PCE, MLP and LSTM-RNN


Conclusions and Future Work Thank you!

Conclusions

Physics-based NN architecture outperforms traditional NN methodsand PCE

Using Tree-Structure LSTM gate can train on the noisy daily data

Qualitatively similar sensitivity results compared to PCE

Future Work

Employ the surrogate for calibration and optimal experimental design

Analyze multiple sites

Spatio-temporal physics-based RNN

Incorporate low-rank tensor structure into RNN


ELM Simulation Details

50°N

40°N

30°N

50°N

40°N

30°N

110°W 80°W

(i, j) (i+ 1, j) (i+ 2, j)

(i, j + 1)

(i+ 1, j + 1)

(i+ 2, j + 1)

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

ACMf(TM , Tm,

BTRAN,FSDS)




SOMDecomposition

f(TM , Tm)


fpgleaf C:N

leaf C:Nfroot C:N

livewd C:Nbr mr

q10 mrrg frac

cstor tau

froot leafstem leaf

croot stemf livewd



r mortk l(1,2,3)


flab(lf,fr)fpi

q10 hr

q10 hrk som(1,2,3,4)


soil4ci

GPP Rg, Rm

NPP

Litter1-3

SOM1-4

HR

Leaf(LAI)

stem

root

NEE

drivers cell (i+ 1, j + 1)

dayk−1 dayk dayk+1

drivers cell (i, j + 1)


drivers cell (i+ 2, j + 1)


drivers cell (i+ 1, j)


drivers cell (i, j)


drivers cell (i+ 2, j)


Global Sensitivity Analysis

Y = f (X1,X2,X3, ...,XN)

Total Variance decomposition (normalized)

Var(Y ) = E [Var(Y |X )] + Var(E [Y |X ])

If Xi are independent: V (Y ) =N∑i=1

Vi +∑

1≤i≤j≤NVij + ..+ V1,..,p

Use Sobol indices → QOI’s variance to be decomposed based onvariances of inputs

Sobol Indices

Si =VarXi

(E (Y |Xi ))

Var(Y )First Sobol Indices

Sij =Varij

Var(Y )Second Order Sobol Indices

Method

PCE allows for analytical computationof Si

ML needs Monte Carlo Integration tocompute Si

Surrogate Modeling:Tree LSTM

LSTM Equations

fjk = σ(Wf xj +∑

Uklhjl)

ij = σ(Wixj +∑

Ulhjl)

oj = σ(Woxj + Ulhjl)

uj = tanh(Wcxj + Ulhjl)

cj =∑

fjl � cjl + ij � uj

hj = σj � tanh(cj)

(1)

physics-informed recurrent neural networks for...

Documents