Global Sensitivity Analysis for Systemswith Independent and/or Correlated
Inputs
Herschel Rabitz and Genyuan Li
Department of ChemistryPrinceton University
Princeton, NJ 08544, USA
Outline
• General formulation of unique HDMR component functions with independentand/or correlated variables
• HDMR formulation specific for independent variables
• HDMR formulation general for independent and/or correlated variables
• Numerical determination of HDMR component functions• Extended bases• D-MORPH (Diffeomorphic Modulation under Observable Response
Preserving Homotopy) regression• Cost function related to hierarchical orthogonality• A test example
• Global sensitivity analysis for independent and/or correlated inputs
• Covariance decomposition of the output variance
• Structural (independent) and correlative sensitivity analysis
• Illustrations
• Conclusions
HDMR formulation specific for independent variables
f(x) = f0 +n∑
i=1
fi(xi) +∑
1≤i<j≤n
fij(xi, xj) + · · ·+ f12...n(x1, x2, . . . , xn) = f0 +2n−1∑j=1
fpj(xpj
)
Using the vanishing condition or equivalently the mutual orthogonality condition∫wis(xis)fi1i2...il(xi1 , xi2 , . . . , xil)dxis = 0, is ∈ {i1, i2, . . . , il}∫
w(x)fi1i2···ik(xi1 , xi2 , . . . , xik)fj1j2···jl(xj1 , xj2 , . . . , xjl
)dx = 0, {i1, i2, · · · , ik} 6= {j1, j2, · · · , jl}
the HDMR component functions are uniquely defined as
f0 =
∫ n∏i=1
wi(xi)f(x)dx
fi(xi) =
∫ n∏k=1,k 6=i
wk(xk)f(x)dx−i − f0
fij(xi, xj) =
∫ n∏k=1,k 6=i,j
wk(xk)f(x)dx−ij − fi(xi)− fj(xj)− f0
. . .
where the pdf and conditional pdf’s are products of the marginal pdf wi(xi)’s.
General formulation of HDMR component functions
The HDMR expansion is expressed in a compact form
f(x) =∑u⊆n
fu(xu)
where u ⊆ {1, 2, . . . , n} and u ⊆ n represents u ⊆ {1, 2, . . . , n} (∅ ∈ u and f0 ∈ fu(xu)).
Hooker* proved that if the support of w(x) is grid closed, under a relaxed vanishing condition
∀u ⊆ n, ∀i ∈ u,
∫fu(xu)w(x)dx−udxi =
∫fu(xu)wu(xu)dxi = 0.
or equivalent hierarchical orthogonality condition
∀v ⊂ u, ∀gv :
∫fu(xu)gv(xv)w(x)dx = 〈fu(xu), gv(xv)〉 = 0,
the minimization of the squared error
{fu(xu | u ⊆ n)} = argmin{gu∈L2(Ru),u⊆n}
∫ (∑u⊆n
gu(xu)− f(x)
)2
w(x)dx
has a unique solution for HDMR component functions.
General formulation of HDMR component functions**
The general formulas of HDMR component functions obtained by using the relaxed vanishing condition
f0 =
∫f(x)w(x)dx
fi(xi) =
∫f(x)w−i(x−i)dx−i − f0 −
∑{i}⊂u⊆n
∫fu(xu)w−i(x−i)dx−i
fij(xi, xj) =
∫f(x)w−ij(x−ij)dx−ij − f0 − fi(xi)− fj(xj)−
∑u⊆n
{i,j}T
u 6=∅
∫fu(xu)w−ij(x−ij)dx−ij
· · ·
• Component functions are coupled one another, and their explicit formulas cannot be determinedsequentially starting from f0.
• For independent variables the last term vanishes and the formulas reduce to those for independentvariables.
*G.Hooker,“Generalized functional ANOVA diagnostics for high dimensional functions of dependentvariables”, J. Comput. Graph. Stat. Soc., 16(3), 709-732(2007).**G. Li, H. Rabitz, “General formulation of HDMR component functions with independent and cor-related variables”, J. Math. Chem., 50(1),99-130(2012).
Numerical determination of HDMR component functions
A practical numerical method is developed to determine the unique HDMR compo-nent functions.
• This numerical method is based on minimization of squared error under hierar-chical orthogonality condition
• Two previously developed approaches
• Extended bases*
• D-MORPH regression**
were used to develop this numerical method
*G. Li, et al., “Random sampling-high dimensional model representation (RS-HDMR)and orthogonality of its different order component functions” J. Phys. Chem. A,110(7), 2474-2485(2006).
**G. Li, H. Rabitz, “D-MORPH regression: application to modeling with unknownparameters more than observation data”, J. Math. Chem., 48(4),1010-1035(2010).
Extended bases
In practice, HDMR component functions are approximated by expansions in suitable basis functions{ϕ} (orthonormal polynomials, splines, etc.). The sufficient condition for hierarchical orthogonalityis that the space spanned by the basis functions for any lower order component function is a normalsubspace of the space spanned by the basis functions of the nested higher order component functions.
fi(xi) ≈k∑
r=1
α(0)ir ϕi
r(xi),
fij(xi, xj) ≈k∑
r=1
[α(ij)ir ϕi
r(xi) + α(ij)jr ϕj
r(xj)] +l∑
p=1
l∑q=1
β(0)ijpq ϕi
p(xi)ϕjq(xj),
fijk(xi, xj, xk) ≈k∑
r=1
[α(ijk)ir ϕi
r(xi) + α(ijk)jr ϕj
r(xj) + α(ijk)kr ϕk
r(xk)]
+l∑
p=1
l∑q=1
[β(ijk)ijpq ϕi
p(xi)ϕjq(xj) + β(ijk)ik
pq ϕip(xi)ϕ
kq(xk) + β(ijk)jk
pq ϕjp(xj)ϕ
kq(xk)]
+m∑
p=1
m∑q=1
m∑r=1
γ(0)ijkpqr ϕi
p(xi)ϕjq(xj)ϕ
kr(xk),
Extended bases
f(x) ≈ f0 +n∑
i=1
k∑r=1
(α(0)ir +
n∑j=1j 6=i
α(ij)ir +
n∑j<k=1j,k 6=i
α(ijk)ir )ϕi
r(xi)
+∑
1≤i<j≤n
l∑p=1
l∑q=1
(β(0)ijpq +
n∑k=1k 6=i,j
β(ijk)ijpq )ϕi
p(xi)ϕjq(xj) +
∑1≤i<j<k≤n
m∑p=1
m∑q=1
m∑r=1
γ(0)ijkpqr ϕi
p(xi)ϕjq(xj)ϕ
kr(xk)
It can be written in vector form for all of N data
φ(x(s))Tc = f(x(s))− f0, (s = 1, 2, . . . , N)
or in matrix formΦc = b
The coefficients c can be obtained by solving the normal equation
ΦTΦc = ΦTb.
Due to extended bases, some equations are identical. These redundant equations can be removed togive an underdetermined linear algebraic equation system.
D-MORPH regression
Removing the redundant equations gives an underdetermined equation system
Ac = d.
There are an infinite number of solutions which compose a convex set M:
c = A+d + (I− A+A)u
D-MORPH regression searches for a solution satisfying an extra requirement by considering an explo-ration path c(s) within M with s in [0,∞)
dc(s)
ds= (I− A+A)
u(s)
ds= (I− A+A)v(s) = Pv(s)
v(s) may be freely chosen to not only enable broad choices for exploring c(s), but to also continuouslyreduce a defined cost K(c(s)). If the free function vector v(s) is chosen as
v(s) = −∂K(c(s))
∂c,
thendK(c(s))
ds≤ 0. ∀s
D-MORPH regression
The solution which minimizes K is given by
c∞ = lims→∞
c(s)
When the cost function is defined as a quadratic form in c
K =1
2cT Bc
with B being symmetric and non-negative definite, then
dc(s)
ds= −PBc(s)
Suppose the singular value decomposition of PB is given by
PB = U
[Sr 00 0
]V T
The analytical form of c∞ isc∞ = Vt−r(U
Tt−rVt−r)
−1UTt−rA
+d
where r is the number of nonzero singular values, t is the dimension of c, Ut−r, and Vt−r are the lastt− r columns of U and V .
Cost function related to hierarchical orthogonality
1. First order component function fi(xi) is orthogonal to f0∫f0fi(xi)wi(xi)dxi = 0 ⇒
∫fi(xi)wi(xi)dxi = 0, (i = 1, 2, . . . , n)
When fi(xi) is represented as a linear combination of basis functions rij(xi)(j = q +
1, . . . , q + k),∫ q+k∑j=q+1
cjrij(xi)wi(xi)dxi ≈
q+k∑j=q+1
cj
(N∑
s=1
rij(x
(s)i )/N
)= Sr(xi)
Tci = 0.
The corresponding cost function for fi(xi) is set to be
Ki =1
2(ci)TSr(xi)Sr(xi)
Tci =1
2(ci)TBici
where Bi is a k × k symmetric and non-negative definite matrix.
Cost function related to hierarchical orthogonality
2. Second order component function fij(xi, xj) is orthogonal to f0, fi(xi), fj(xj)
If fi(xi) =∑k
u=1 ciur
iu(xi), fj(xj) =
∑kv=1 cj
vrjv(xj), the above orthogonalities are equiv-
alent to ∫fij(xi, xj)wij(xi, xj)dxidxj = 0,∫riu(xi)fij(xi, xj)wij(xi, xj)dxidxj = 0, (u = 1, 2, . . . , k)∫
rjv(xj)fij(xi, xj)wij(xi, xj)dxidxj = 0, (v = 1, 2, . . . , k)
The cost function reflecting the above orthogonalities is defined as
Kij =1
2(cij)TSr(xi, xj)Sr(xi, xj)
Tcij =1
2(cij)TBijcij
where Bij is a symmetric and non-negative definite matrix.
Cost function related to hierarchical orthogonality
The total cost function for the 3rd order HDMR expansion is set to be
K =n∑
i=1
Ki +∑
1≤i<j≤n
Kij +∑
1≤i<j<k≤n
Kijk =1
2cTBc
where c consists of all the unknown coefficients. The symmetric and non-negativedefinite matrix B is
B =
0BBBBBBBBBBBBBBBB@
B1
. . .
Bn
B12
. . .
B(n−1)n
B123
. . .
B(n−2)(n−1)n
1CCCCCCCCCCCCCCCCA
A test example
Consider the following nonlinear function with three variables
f(x) = g1(x1, x2) + g2(x2) + g3(x3),
where
g1(x1, x2) = g1a(x1)g1b(x2) = [a1(x1 − µ1) + a0][b1(x2 − µ2) + b0],
g2(x2) = c2(x2 − µ2)2 + c1(x2 − µ2) + c0,
g3(x3) = d3(x3 − µ3)3 + d2(x3 − µ3)
2 + d1(x3 − µ3) + d0
with a multivariate normal distribution
w(x) =1
(2π)3/2 | Σ |1/2exp
(−1
2(x− µ)TΣ−1(x− µ)
),
where µ = (µ1, µ2, µ3) is the expected value of x, Σ is the covariance matrix of x
Σ =
σ21 σ12 0
σ12 σ22 0
0 0 σ23
=
σ21 ρ12σ1σ2 0
ρ12σ1σ2 σ22 0
0 0 σ23
i.e., x1 and x2 are correlated, but x3 is independent.
A test example
Analytical formulas of the unique HDMR component functions for f(x):
f0 = a1b1ρ12σ1σ2 + a0b0 + c2σ22 + c0 + d2σ
23 + d0,
f1(x1) = a1b1σ2
σ1
ρ12
ρ212 + 1
(x1 − µ1)2 + a1b0(x1 − µ1)− a1b1σ1σ2
ρ12
ρ212 + 1
,
f2(x2) =
[a1b1
σ1
σ2
ρ12
ρ212 + 1
+ c2
](x2 − µ2)
2
+ (a0b1 + c1)(x2 − µ2)− a1b1σ1σ2ρ12
ρ212 + 1
− c2σ22,
f3(x3) = d3(x3 − µ3)3 + d2(x3 − µ3)
2 + d1(x3 − µ3)− d2σ23,
f12(x1, x2) = −a1b1σ2
σ1
ρ12
ρ212 + 1
(x1 − µ1)2 + a1b1(x1 − µ1)(x2 − µ2)
− a1b1σ1
σ2
ρ12
ρ212 + 1
(x2 − µ2)2 − a1b1ρ12σ1σ2
ρ212 − 1
ρ212 + 1
,
f13(x1, x3) = 0,
f23(x2, x3) = 0.
A test example
300 points of x were generated according to the multivariate normal distribution with
a0 = 1 a1 = 2 b0 = 2 b1 = 3 c0 = 3 c1 = 1 c2 = 2d0 = 1 d1 = 2 d2 = 2 d3 = 3µ1 = 0.5 µ2 = 0.5 µ3 = 0.5 σ1 = 0.2 σ2 = 0.2 σ3 = 0.18 ρ12 = 0.6
Using F -test, the 2nd order HDMR expansion
f(x) = f0 +3∑
i=1
fi(xi) + f12(x1, x2)
= f0 +3∑
i=1
3∑r=1
α(0)ir ϕi
r(xi) +3∑
r=1
[α(12)1r ϕ1
r(x1) + α(12)2r ϕ2
r(x2)]
+3∑
p=1
3∑q=1
β(0)12pq ϕ1
p(x1)ϕ2q(x2)
with 24 unknown parameters were used where ϕir(xi) are orthonormal polynomials.
A test example
1. Data f(x) without random error
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
0 0.2 0.4 0.6 0.8 1
f 1(x
1)
x1
Analytical solutionLeast-squares solution
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
0 0.2 0.4 0.6 0.8 1
f 2(x
2)
x2
Analytical solutionLeast-squares solution
-1
-0.5
0
0.5
1
1.5
2
0 0.2 0.4 0.6 0.8 1
f 3(x
3)
x3
Analytical solutionLeast-squares solution
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
0 0.2 0.4 0.6 0.8 1
f 1(x
1)
x1
Analytical solutionD-MORPH solution
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
3.5
0 0.2 0.4 0.6 0.8 1
f 2(x
2)
x2
Analytical solutionD-MORPH solution
-1
-0.5
0
0.5
1
1.5
2
0 0.2 0.4 0.6 0.8 1
f 3(x
3)
x3
Analytical solutionD-MORPH solution
Figure 1. Comparison of fi(xi) (i = 1, 2, 3) obtained analytically and numerically byleast-squares regression and D-MORPH regression with extended bases
A test example
-2
-1
0
1
2
3
-0.8 -0.6 -0.4 -0.2 0 0.2
Leas
t-sq
ures
f 12(
x 1,x
2)
Analytical f12(x1,x2)
Least-squares solution
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-0.8 -0.6 -0.4 -0.2 0 0.2
D-M
OR
PH
f 12(
x 1,x
2)
Analytical f12(x1,x2)
D-MORPH solution
Figure 2. Comparison of f12(x1, x2) obtained analytically and numerically by least-squares regression and D-MORPH regression with extended bases
A test example
2. Data f(x) with random error: signal to noise ratio (Var(f(x))/σ2)= 100
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
0 0.2 0.4 0.6 0.8 1
f 1(x
1)
x1
Signal to noise ratio: 100Signal to noise ratio: 100
Analytical solutionD-MORPH solution
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
3.5
0 0.2 0.4 0.6 0.8 1
f 2(x
2)
x2
Signal to noise ratio: 100
Analytical solutionD-MORPH solution
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
3.5
0 0.2 0.4 0.6 0.8 1
f 2(x
2)
x2
Signal to noise ratio: 100
Analytical solutionD-MORPH solution
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-0.8 -0.6 -0.4 -0.2 0 0.2D
-MO
RP
H f 1
2(x 1
,x2)
Analytical f12(x1,x2)
Signal to noise ratio: 100
D-MORPH solution
Figure 3. Comparison of fi(xi) and f12(x1, x2) obtained analytically and numericallyby D-MORPH regression with extended bases for data with random error
Covariance decomposition of the output variance
• For independent and correlated inputs, Var(y) can be uniquely decomposed as
Var(y) = E[(y − E(y))2] =
∫w(x)(y − f0)
2dx = 〈y − f0, y − f0〉
= 〈2n−1∑j=1
fpj, y − f0〉 =
2n−1∑j=1
Cov(fpj, y)
and each covariance consists of two terms: structural (independent) and correlative contributionsof inputs
Cov(fpj, y) = 〈fpj
,
2n−1∑k=1
fpk〉 = 〈fpj
, fpj〉+ 〈fpj
,2n−1∑
k=1,k 6=j
fpk〉
= Var(fpj) +
2n−1∑k=1,k 6=j
Cov(fpj, fpk
)
• For independent inputs, fpj’s are mutually orthogonal and Cov(fpj
, fpk) = 0, then the covariance
decomposition of Var(y) reduces to variance decomposition
Var(y) =2n−1∑j=1
Var(fpj)
Structural (independent) and correlative sensitivity analysis (SCSA)*
• Three sensitivity indices are defined as
Spj= Cov(fpj
, y)/Var(y) ≈N∑
s=1
fpj(x(s)
pj)(y(s) − y)/
N∑s=1
(y(s) − y)2
Sapj
= Var(fpj)/Var(y) ≈
N∑s=1
(fpj(x(s)
pj))2/
N∑s=1
(y(s) − y)2
Sbpj
=2n−1∑
k=1,k 6=j
Cov(fpj, fpk
)/Var(y) = Spj− Sa
pj
• Spj, Sa
pj, Sb
pjgive the total, structural (independent) and correlative contributions for xpj
, respec-tively
• All three indices need to be considered to quantitatively determine the relative importance.
• For truncated HDMR expansion, the sensitivity analysis is reliable if∑np
j=1 Spj≈ 1
• For independent inputs, Sbpj
= 0, then Spj= Sa
pjreducing to a single index Spj
consistent withvariance-based methods
*G. Li, H. Rabitz, et al., J. Phys. Chem. A, 114, 6022(2010).
Illustrations and applications: Example 1
1. A linear function with multinormal distribution
y = f(x) =n∑
i=1
aixi
w(x) =1
(2π)n/2 | Σ |1/2exp
(−1
2(x− µ)TΣ−1(x− µ)
)The analytical formulas of fi(xi) and Si, S
ai , Sb
i can be obtained
f(x) = f0 +n∑
i=1
fi(xi) =n∑
i=1
aixi +n∑
i=1
ai(xi − xi)
Var(y) =n∑
i=1
a2i σ
2i + 2
∑1≤i<j≤n
aiajσij = aT Σa
Var(fi(xi)) = a2i σ
2i Cov(fi(xi), fj(xj)) = aiajσij
Sai = a2
i σ2i /Var(y), Sb
i =∑
j=1,j 6=i
aiajσij/Var(y), Si = Sai + Sb
i
i.e., Sai is dependent only on its deterministic contribution (ai) in f(x) and its marginal pdf (σi); Sb
i
is dependent on the product of its deterministic contribution (ai) and other correlated variables (aj’s)and their correlations σij.
Illustrations and applications: Example 1
Consider an exampley = f(x) = x1 + x2 + x3
with the covariance matrix
Σ =
1 0 00 1 ρσ0 ρσ σ2
Var(y) = 1T Σ1 = 2 + σ2 + 2ρσ
Input SCSA method Variance-based method∗
Sai Sb
i Si Si
x11
2+σ2+2ρσ 0 12+σ2+2ρσ
12+σ2+2ρσ
x21
2+σ2+2ρσρσ
2+σ2+2ρσ1+ρσ
2+σ2+2ρσ(1+ρσ)2
2+σ2+2ρσ
x3σ2
2+σ2+2ρσρσ
2+σ2+2ρσσ2+ρσ
2+σ2+2ρσ(σ2+ρσ)2
2+σ2+2ρσ
sum 2+σ2
2+σ2+2ρσ2ρσ
2+σ2+2ρσ 1 2+σ2+2ρσ+ρ2+3ρ2σ2
2+σ2+2ρσ
*S. Kucherenko, S. Tarantola, P. Annoni, “Estimation of global sensitivity indices for models withdependent variables”, Computer Physics Communications, 183, 937-946(2012).
Illustrations and applications: Example 2
2. A nonlinear function with multinormal distribution given above
y = f(x) = g1(x1, x2) + g2(x2) + g3(x3)
= 3x33 + 6x1x2 + 2x2
2 − 2.5x23 + x1 − x2 + 2.25x3 + 3.125
Σ =
0.0400 0.02400.0240 0.0400
0.0324
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
-0.2 0 0.2 0.4 0.6 0.8 1 1.2
x 2
x1
The distribution of (x1, x2)
Illustrations and applications: Example 2
The results are given below:
Input SCSA method Variance-based methodSa Sb S Vi/Var(y)
x1 0.272± 0.046(0.279) 0.165± 0.021(0.168) 0.438± 0.045(0.447) 0.712± 0.065x2 0.301± 0.048(0.303) 0.168± 0.017(0.168) 0.469± 0.048(0.471) 0.738± 0.055x3 0.077± 0.022(0.076) 0.000± 0.026(0.000) 0.077± 0.031(0.076) 0.085± 0.057
(x1, x2) 0.005± 0.003(0.007) 0.000± 0.002(0.000) 0.005± 0.003(0.007) -sum 0.656± 0.048(0.665) 0.333± 0.050(0.335) 0.989± 0.003(1.000) 1.536± 0.103
*The value in the parentheses ( ) is the true value.
• Sbi correctly identify the correlation between x1 and x2, the independence of x3
• f12(x1, x2) is orthogonal to nested f1(x1) and f2(x2)
• The results given by the variance-based methods mix together the structural(independent) and correlative contributions of the inputs, and the informationgiven by Vi/Var(y) can be misleading
Conclusions
• General formulas of HDMR component functions with independent and/or cor-related variables based on hierarchical orthogonality have been constructed .The resultant HDMR component functions are unique. The original formulasof HDMR component functions which are mutually orthogonal are a special casefor independent variables
• The analytical formulas of HDMR component functions for correlated variablesmay not be explicitly determined, but a numerical method based on extendedbases and D-MORPH regression can determine them
• The structural (independent) and correlative sensitivity analysis based on covari-ance decomposition is general and can be applied to systems with independentand/or correlated inputs. The variance-based method is only a special case ofthe covariance-based method for independent inputs
• Using the general HDMR component functions the general sensitivity indices forindependent and /or correlated inputs can be obtained