global sensitivity analysis for systems with independent and/or … · 2012. 4. 16. · • global...

Global Sensitivity Analysis for Systemswith Independent and/or Correlated

Inputs

Herschel Rabitz and Genyuan Li

Department of ChemistryPrinceton University

Princeton, NJ 08544, USA

Outline

• General formulation of unique HDMR component functions with independentand/or correlated variables

• HDMR formulation specific for independent variables

• HDMR formulation general for independent and/or correlated variables

• Numerical determination of HDMR component functions• Extended bases• D-MORPH (Diffeomorphic Modulation under Observable Response

Preserving Homotopy) regression• Cost function related to hierarchical orthogonality• A test example

• Global sensitivity analysis for independent and/or correlated inputs

• Covariance decomposition of the output variance

• Structural (independent) and correlative sensitivity analysis

• Illustrations

• Conclusions

HDMR formulation specific for independent variables

f(x) = f0 +n∑

i=1

fi(xi) +∑

1≤i<j≤n

fij(xi, xj) + · · ·+ f12...n(x1, x2, . . . , xn) = f0 +2n−1∑j=1

fpj(xpj

)

Using the vanishing condition or equivalently the mutual orthogonality condition∫wis(xis)fi1i2...il(xi1 , xi2 , . . . , xil)dxis = 0, is ∈ {i1, i2, . . . , il}∫

w(x)fi1i2···ik(xi1 , xi2 , . . . , xik)fj1j2···jl(xj1 , xj2 , . . . , xjl

)dx = 0, {i1, i2, · · · , ik} 6= {j1, j2, · · · , jl}

the HDMR component functions are uniquely defined as

f0 =

∫ n∏i=1

wi(xi)f(x)dx

fi(xi) =

∫ n∏k=1,k 6=i

wk(xk)f(x)dx−i − f0

fij(xi, xj) =

∫ n∏k=1,k 6=i,j

wk(xk)f(x)dx−ij − fi(xi)− fj(xj)− f0

. . .

where the pdf and conditional pdf’s are products of the marginal pdf wi(xi)’s.

General formulation of HDMR component functions

The HDMR expansion is expressed in a compact form

f(x) =∑u⊆n

fu(xu)

where u ⊆ {1, 2, . . . , n} and u ⊆ n represents u ⊆ {1, 2, . . . , n} (∅ ∈ u and f0 ∈ fu(xu)).

Hooker* proved that if the support of w(x) is grid closed, under a relaxed vanishing condition

∀u ⊆ n, ∀i ∈ u,

∫fu(xu)w(x)dx−udxi =

∫fu(xu)wu(xu)dxi = 0.

or equivalent hierarchical orthogonality condition

∀v ⊂ u, ∀gv :

∫fu(xu)gv(xv)w(x)dx = 〈fu(xu), gv(xv)〉 = 0,

the minimization of the squared error

{fu(xu | u ⊆ n)} = argmin{gu∈L2(Ru),u⊆n}

∫ (∑u⊆n

gu(xu)− f(x)

)2

w(x)dx

has a unique solution for HDMR component functions.

General formulation of HDMR component functions**

The general formulas of HDMR component functions obtained by using the relaxed vanishing condition

f0 =

∫f(x)w(x)dx

fi(xi) =

∫f(x)w−i(x−i)dx−i − f0 −

∑{i}⊂u⊆n

∫fu(xu)w−i(x−i)dx−i

fij(xi, xj) =

∫f(x)w−ij(x−ij)dx−ij − f0 − fi(xi)− fj(xj)−

∑u⊆n

{i,j}T

u 6=∅

∫fu(xu)w−ij(x−ij)dx−ij

· · ·

• Component functions are coupled one another, and their explicit formulas cannot be determinedsequentially starting from f0.

• For independent variables the last term vanishes and the formulas reduce to those for independentvariables.

*G.Hooker,“Generalized functional ANOVA diagnostics for high dimensional functions of dependentvariables”, J. Comput. Graph. Stat. Soc., 16(3), 709-732(2007).**G. Li, H. Rabitz, “General formulation of HDMR component functions with independent and cor-related variables”, J. Math. Chem., 50(1),99-130(2012).

Numerical determination of HDMR component functions

A practical numerical method is developed to determine the unique HDMR compo-nent functions.

• This numerical method is based on minimization of squared error under hierar-chical orthogonality condition

• Two previously developed approaches

• Extended bases*

• D-MORPH regression**

were used to develop this numerical method

*G. Li, et al., “Random sampling-high dimensional model representation (RS-HDMR)and orthogonality of its different order component functions” J. Phys. Chem. A,110(7), 2474-2485(2006).

**G. Li, H. Rabitz, “D-MORPH regression: application to modeling with unknownparameters more than observation data”, J. Math. Chem., 48(4),1010-1035(2010).

Extended bases

In practice, HDMR component functions are approximated by expansions in suitable basis functions{ϕ} (orthonormal polynomials, splines, etc.). The sufficient condition for hierarchical orthogonalityis that the space spanned by the basis functions for any lower order component function is a normalsubspace of the space spanned by the basis functions of the nested higher order component functions.

fi(xi) ≈k∑

r=1

α(0)ir ϕi

r(xi),

fij(xi, xj) ≈k∑

r=1

[α(ij)ir ϕi

r(xi) + α(ij)jr ϕj

r(xj)] +l∑

p=1

l∑q=1

β(0)ijpq ϕi

p(xi)ϕjq(xj),

fijk(xi, xj, xk) ≈k∑

r=1

[α(ijk)ir ϕi

r(xi) + α(ijk)jr ϕj

r(xj) + α(ijk)kr ϕk

r(xk)]

+l∑

p=1

l∑q=1

[β(ijk)ijpq ϕi

p(xi)ϕjq(xj) + β(ijk)ik

pq ϕip(xi)ϕ

kq(xk) + β(ijk)jk

pq ϕjp(xj)ϕ

kq(xk)]

+m∑

p=1

m∑q=1

m∑r=1

γ(0)ijkpqr ϕi

p(xi)ϕjq(xj)ϕ

kr(xk),

Extended bases

f(x) ≈ f0 +n∑

i=1

k∑r=1

(α(0)ir +

n∑j=1j 6=i

α(ij)ir +

n∑j<k=1j,k 6=i

α(ijk)ir )ϕi

r(xi)

+∑

1≤i<j≤n

l∑p=1

l∑q=1

(β(0)ijpq +

n∑k=1k 6=i,j

β(ijk)ijpq )ϕi

p(xi)ϕjq(xj) +

∑1≤i<j<k≤n

m∑p=1

m∑q=1

m∑r=1

γ(0)ijkpqr ϕi

p(xi)ϕjq(xj)ϕ

kr(xk)

It can be written in vector form for all of N data

φ(x(s))Tc = f(x(s))− f0, (s = 1, 2, . . . , N)

or in matrix formΦc = b

The coefficients c can be obtained by solving the normal equation

ΦTΦc = ΦTb.

Due to extended bases, some equations are identical. These redundant equations can be removed togive an underdetermined linear algebraic equation system.

D-MORPH regression

Removing the redundant equations gives an underdetermined equation system

Ac = d.

There are an infinite number of solutions which compose a convex set M:

c = A+d + (I− A+A)u

D-MORPH regression searches for a solution satisfying an extra requirement by considering an explo-ration path c(s) within M with s in [0,∞)

dc(s)

ds= (I− A+A)

u(s)

ds= (I− A+A)v(s) = Pv(s)

v(s) may be freely chosen to not only enable broad choices for exploring c(s), but to also continuouslyreduce a defined cost K(c(s)). If the free function vector v(s) is chosen as

v(s) = −∂K(c(s))

∂c,

thendK(c(s))

ds≤ 0. ∀s

D-MORPH regression

The solution which minimizes K is given by

c∞ = lims→∞

c(s)

When the cost function is defined as a quadratic form in c

K =1

2cT Bc

with B being symmetric and non-negative definite, then

dc(s)

ds= −PBc(s)

Suppose the singular value decomposition of PB is given by

PB = U

[Sr 00 0

]V T

The analytical form of c∞ isc∞ = Vt−r(U

Tt−rVt−r)

−1UTt−rA

+d

where r is the number of nonzero singular values, t is the dimension of c, Ut−r, and Vt−r are the lastt− r columns of U and V .

Cost function related to hierarchical orthogonality

1. First order component function fi(xi) is orthogonal to f0∫f0fi(xi)wi(xi)dxi = 0 ⇒

∫fi(xi)wi(xi)dxi = 0, (i = 1, 2, . . . , n)

When fi(xi) is represented as a linear combination of basis functions rij(xi)(j = q +

1, . . . , q + k),∫ q+k∑j=q+1

cjrij(xi)wi(xi)dxi ≈

q+k∑j=q+1

cj

(N∑

s=1

rij(x

(s)i )/N

)= Sr(xi)

Tci = 0.

The corresponding cost function for fi(xi) is set to be

Ki =1

2(ci)TSr(xi)Sr(xi)

Tci =1

2(ci)TBici

where Bi is a k × k symmetric and non-negative definite matrix.


2. Second order component function fij(xi, xj) is orthogonal to f0, fi(xi), fj(xj)

If fi(xi) =∑k

u=1 ciur

iu(xi), fj(xj) =

∑kv=1 cj

vrjv(xj), the above orthogonalities are equiv-

alent to ∫fij(xi, xj)wij(xi, xj)dxidxj = 0,∫riu(xi)fij(xi, xj)wij(xi, xj)dxidxj = 0, (u = 1, 2, . . . , k)∫

rjv(xj)fij(xi, xj)wij(xi, xj)dxidxj = 0, (v = 1, 2, . . . , k)

The cost function reflecting the above orthogonalities is defined as

Kij =1

2(cij)TSr(xi, xj)Sr(xi, xj)

Tcij =1

2(cij)TBijcij

where Bij is a symmetric and non-negative definite matrix.


The total cost function for the 3rd order HDMR expansion is set to be

K =n∑

i=1

Ki +∑

1≤i<j≤n

Kij +∑

1≤i<j<k≤n

Kijk =1

2cTBc

where c consists of all the unknown coefficients. The symmetric and non-negativedefinite matrix B is

B =

0BBBBBBBBBBBBBBBB@

B1

. . .

Bn

B12

. . .

B(n−1)n

B123

. . .

B(n−2)(n−1)n

1CCCCCCCCCCCCCCCCA

A test example

Consider the following nonlinear function with three variables

f(x) = g1(x1, x2) + g2(x2) + g3(x3),

where

g1(x1, x2) = g1a(x1)g1b(x2) = [a1(x1 − µ1) + a0][b1(x2 − µ2) + b0],

g2(x2) = c2(x2 − µ2)2 + c1(x2 − µ2) + c0,

g3(x3) = d3(x3 − µ3)3 + d2(x3 − µ3)

2 + d1(x3 − µ3) + d0

with a multivariate normal distribution

w(x) =1

(2π)3/2 | Σ |1/2exp

(−1

2(x− µ)TΣ−1(x− µ)

),

where µ = (µ1, µ2, µ3) is the expected value of x, Σ is the covariance matrix of x

Σ =

σ21 σ12 0

σ12 σ22 0

0 0 σ23

=

σ21 ρ12σ1σ2 0

ρ12σ1σ2 σ22 0

0 0 σ23

i.e., x1 and x2 are correlated, but x3 is independent.

A test example

Analytical formulas of the unique HDMR component functions for f(x):

f0 = a1b1ρ12σ1σ2 + a0b0 + c2σ22 + c0 + d2σ

23 + d0,

f1(x1) = a1b1σ2

σ1

ρ12

ρ212 + 1

(x1 − µ1)2 + a1b0(x1 − µ1)− a1b1σ1σ2

ρ12

ρ212 + 1

,

f2(x2) =

[a1b1

σ1

σ2

ρ12

ρ212 + 1

+ c2

](x2 − µ2)

2

+ (a0b1 + c1)(x2 − µ2)− a1b1σ1σ2ρ12

ρ212 + 1

− c2σ22,

f3(x3) = d3(x3 − µ3)3 + d2(x3 − µ3)

2 + d1(x3 − µ3)− d2σ23,

f12(x1, x2) = −a1b1σ2

σ1

ρ12

ρ212 + 1

(x1 − µ1)2 + a1b1(x1 − µ1)(x2 − µ2)

− a1b1σ1

σ2

ρ12

ρ212 + 1

(x2 − µ2)2 − a1b1ρ12σ1σ2

ρ212 − 1

ρ212 + 1

,

f13(x1, x3) = 0,

f23(x2, x3) = 0.

A test example

300 points of x were generated according to the multivariate normal distribution with

a0 = 1 a1 = 2 b0 = 2 b1 = 3 c0 = 3 c1 = 1 c2 = 2d0 = 1 d1 = 2 d2 = 2 d3 = 3µ1 = 0.5 µ2 = 0.5 µ3 = 0.5 σ1 = 0.2 σ2 = 0.2 σ3 = 0.18 ρ12 = 0.6

Using F -test, the 2nd order HDMR expansion

f(x) = f0 +3∑

i=1

fi(xi) + f12(x1, x2)

= f0 +3∑

i=1

3∑r=1

α(0)ir ϕi

r(xi) +3∑

r=1

[α(12)1r ϕ1

r(x1) + α(12)2r ϕ2

r(x2)]

+3∑

p=1

3∑q=1

β(0)12pq ϕ1

p(x1)ϕ2q(x2)

with 24 unknown parameters were used where ϕir(xi) are orthonormal polynomials.

A test example

1. Data f(x) without random error

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

0 0.2 0.4 0.6 0.8 1

f 1(x

1)

x1

Analytical solutionLeast-squares solution

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

0 0.2 0.4 0.6 0.8 1

f 2(x

2)

x2


-1

-0.5

0

0.5

1

1.5

2

0 0.2 0.4 0.6 0.8 1

f 3(x

3)

x3


-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

0 0.2 0.4 0.6 0.8 1

f 1(x

1)

x1

Analytical solutionD-MORPH solution

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

0 0.2 0.4 0.6 0.8 1

f 2(x

2)

x2


-1

-0.5

0

0.5

1

1.5

2

0 0.2 0.4 0.6 0.8 1

f 3(x

3)

x3


Figure 1. Comparison of fi(xi) (i = 1, 2, 3) obtained analytically and numerically byleast-squares regression and D-MORPH regression with extended bases

A test example

-2

-1

0

1

2

3

-0.8 -0.6 -0.4 -0.2 0 0.2

Leas

t-sq

ures

f 12(

x 1,x

2)

Analytical f12(x1,x2)

Least-squares solution

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

-0.8 -0.6 -0.4 -0.2 0 0.2

D-M

OR

PH

f 12(

x 1,x

2)


D-MORPH solution

Figure 2. Comparison of f12(x1, x2) obtained analytically and numerically by least-squares regression and D-MORPH regression with extended bases

A test example

2. Data f(x) with random error: signal to noise ratio (Var(f(x))/σ2)= 100

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

0 0.2 0.4 0.6 0.8 1

f 1(x

1)

x1

Signal to noise ratio: 100Signal to noise ratio: 100


-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

0 0.2 0.4 0.6 0.8 1

f 2(x

2)

x2

Signal to noise ratio: 100


-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

0 0.2 0.4 0.6 0.8 1

f 2(x

2)

x2



-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

-0.8 -0.6 -0.4 -0.2 0 0.2D

-MO

RP

H f 1

2(x 1

,x2)



D-MORPH solution

Figure 3. Comparison of fi(xi) and f12(x1, x2) obtained analytically and numericallyby D-MORPH regression with extended bases for data with random error

Covariance decomposition of the output variance

• For independent and correlated inputs, Var(y) can be uniquely decomposed as

Var(y) = E[(y − E(y))2] =

∫w(x)(y − f0)

2dx = 〈y − f0, y − f0〉

= 〈2n−1∑j=1

fpj, y − f0〉 =

2n−1∑j=1

Cov(fpj, y)

and each covariance consists of two terms: structural (independent) and correlative contributionsof inputs

Cov(fpj, y) = 〈fpj

,

2n−1∑k=1

fpk〉 = 〈fpj

, fpj〉+ 〈fpj

,2n−1∑

k=1,k 6=j

fpk〉

= Var(fpj) +

2n−1∑k=1,k 6=j

Cov(fpj, fpk

)

• For independent inputs, fpj’s are mutually orthogonal and Cov(fpj

, fpk) = 0, then the covariance

decomposition of Var(y) reduces to variance decomposition

Var(y) =2n−1∑j=1

Var(fpj)

Structural (independent) and correlative sensitivity analysis (SCSA)*

• Three sensitivity indices are defined as

Spj= Cov(fpj

, y)/Var(y) ≈N∑

s=1

fpj(x(s)

pj)(y(s) − y)/

N∑s=1

(y(s) − y)2

Sapj

= Var(fpj)/Var(y) ≈

N∑s=1

(fpj(x(s)

pj))2/

N∑s=1

(y(s) − y)2

Sbpj

=2n−1∑

k=1,k 6=j

Cov(fpj, fpk

)/Var(y) = Spj− Sa

pj

• Spj, Sa

pj, Sb

pjgive the total, structural (independent) and correlative contributions for xpj

, respec-tively

• All three indices need to be considered to quantitatively determine the relative importance.

• For truncated HDMR expansion, the sensitivity analysis is reliable if∑np

j=1 Spj≈ 1

• For independent inputs, Sbpj

= 0, then Spj= Sa

pjreducing to a single index Spj

consistent withvariance-based methods

*G. Li, H. Rabitz, et al., J. Phys. Chem. A, 114, 6022(2010).

Illustrations and applications: Example 1

1. A linear function with multinormal distribution

y = f(x) =n∑

i=1

aixi

w(x) =1

(2π)n/2 | Σ |1/2exp

(−1

2(x− µ)TΣ−1(x− µ)

)The analytical formulas of fi(xi) and Si, S

ai , Sb

i can be obtained

f(x) = f0 +n∑

i=1

fi(xi) =n∑

i=1

aixi +n∑

i=1

ai(xi − xi)

Var(y) =n∑

i=1

a2i σ

2i + 2

∑1≤i<j≤n

aiajσij = aT Σa

Var(fi(xi)) = a2i σ

2i Cov(fi(xi), fj(xj)) = aiajσij

Sai = a2

i σ2i /Var(y), Sb

i =∑

j=1,j 6=i

aiajσij/Var(y), Si = Sai + Sb

i

i.e., Sai is dependent only on its deterministic contribution (ai) in f(x) and its marginal pdf (σi); Sb

i

is dependent on the product of its deterministic contribution (ai) and other correlated variables (aj’s)and their correlations σij.


Consider an exampley = f(x) = x1 + x2 + x3

with the covariance matrix

Σ =

1 0 00 1 ρσ0 ρσ σ2

Var(y) = 1T Σ1 = 2 + σ2 + 2ρσ

Input SCSA method Variance-based method∗

Sai Sb

i Si Si

x11

2+σ2+2ρσ 0 12+σ2+2ρσ

12+σ2+2ρσ

x21

2+σ2+2ρσρσ

2+σ2+2ρσ1+ρσ

2+σ2+2ρσ(1+ρσ)2

2+σ2+2ρσ

x3σ2

2+σ2+2ρσρσ

2+σ2+2ρσσ2+ρσ

2+σ2+2ρσ(σ2+ρσ)2

2+σ2+2ρσ

sum 2+σ2

2+σ2+2ρσ2ρσ

2+σ2+2ρσ 1 2+σ2+2ρσ+ρ2+3ρ2σ2

2+σ2+2ρσ

*S. Kucherenko, S. Tarantola, P. Annoni, “Estimation of global sensitivity indices for models withdependent variables”, Computer Physics Communications, 183, 937-946(2012).


2. A nonlinear function with multinormal distribution given above

y = f(x) = g1(x1, x2) + g2(x2) + g3(x3)

= 3x33 + 6x1x2 + 2x2

2 − 2.5x23 + x1 − x2 + 2.25x3 + 3.125

Σ =

0.0400 0.02400.0240 0.0400

0.0324

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

-0.2 0 0.2 0.4 0.6 0.8 1 1.2

x 2

x1

The distribution of (x1, x2)


The results are given below:

Input SCSA method Variance-based methodSa Sb S Vi/Var(y)

x1 0.272± 0.046(0.279) 0.165± 0.021(0.168) 0.438± 0.045(0.447) 0.712± 0.065x2 0.301± 0.048(0.303) 0.168± 0.017(0.168) 0.469± 0.048(0.471) 0.738± 0.055x3 0.077± 0.022(0.076) 0.000± 0.026(0.000) 0.077± 0.031(0.076) 0.085± 0.057

(x1, x2) 0.005± 0.003(0.007) 0.000± 0.002(0.000) 0.005± 0.003(0.007) -sum 0.656± 0.048(0.665) 0.333± 0.050(0.335) 0.989± 0.003(1.000) 1.536± 0.103

*The value in the parentheses ( ) is the true value.

• Sbi correctly identify the correlation between x1 and x2, the independence of x3

• f12(x1, x2) is orthogonal to nested f1(x1) and f2(x2)

• The results given by the variance-based methods mix together the structural(independent) and correlative contributions of the inputs, and the informationgiven by Vi/Var(y) can be misleading

Conclusions

• General formulas of HDMR component functions with independent and/or cor-related variables based on hierarchical orthogonality have been constructed .The resultant HDMR component functions are unique. The original formulasof HDMR component functions which are mutually orthogonal are a special casefor independent variables

• The analytical formulas of HDMR component functions for correlated variablesmay not be explicitly determined, but a numerical method based on extendedbases and D-MORPH regression can determine them

• The structural (independent) and correlative sensitivity analysis based on covari-ance decomposition is general and can be applied to systems with independentand/or correlated inputs. The variance-based method is only a special case ofthe covariance-based method for independent inputs

• Using the general HDMR component functions the general sensitivity indices forindependent and /or correlated inputs can be obtained

global sensitivity analysis for systems with independent and/or … · 2012. 4. 16. · • global...

Documents