statistical inferences by gaussian markov random …kazu/tanaka-usui...10 december, 2008 cimca2008...

10 December, 200810 December, 2008 CIMCA2008 (Vienna)CIMCA2008 (Vienna) 11

Statistical Inferences by Gaussian Markov Statistical Inferences by Gaussian Markov Random Fields on Complex NetworksRandom Fields on Complex Networks

Kazuyuki Tanaka, Kazuyuki Tanaka, TakafumiTakafumi UsuiUsui,,MunekiMuneki Yasuda Yasuda

Graduate School of Information Sciences,Graduate School of Information Sciences,Tohoku UniversityTohoku University


Bayesian Network and Bayesian Network and Graphical modelGraphical model

Image Processing

Regular Graph

Code TheoryRandom Graph

Bipartite Graph

Complete Graph

Data Mining

Machine Learning

Probabilistic Inference

Hypergraph

Bayesian networks are Bayesian networks are formulated for formulated for statistical inferences as statistical inferences as probabilistic models on probabilistic models on various networks. various networks. The performances The performances sometimes depend on sometimes depend on the statistical the statistical properties in the properties in the network structures.network structures.

10 December, 2008 CIMCA2008 (Vienna) 3

Recently, various kinds of networks and their stochastically generating models are interested in the

applications of statistical inferences.

Complex Networks

id i vertex of degree the:

∑∈

∝Vi

diddP ,)( δ

Degree Distribution

1

23

4

5{1,2}{2,3}

{3,4}{2,4}

{3,5}

11 =d 15 =d33 =d

24 =d32 =d

)(dP

1 2 3 4 50 d

:Set of all the vertices}5,4,3,2,1{=V{ }}5,3{},4,3{},4,2{},3,2{},2,1{=E :Set of all the edges

The degree of each vertex plays an important role for the statistical properties in the structures of networks.Networks are classified by using the degree distributions.

(The degree of vertex is the number of edges connected to the vertex)


Degree Distribution in Complex Networks

γ−ddP ~)(

Poisson Distribution

Power Law DistributionScale Free Network:

Random Network:

0

0.1

0.2

0.3

0.4

0.5

0.6

0 20 40 60 80 100 120

P(k)

k

scale-freerandom

d

P(d)

Random Network

Scale Free Network

It is known that the degree distributions of random networks areaccording to the Poisson distributions. The scale free networks have some hub-vertices, their degree distributions are given by power law distributions.

d

ddP ρ

!1~)(

10-4

10-3

10-2

10-1

100

100 101 102 103P(

k)

k

scale-freerandom

dP(

d)


Purpose of the present talk

In the present paper, we analyze the statistical performance of the Bayesian inferences on some complex networks including scale free networks. We adopt the Gauss Markov random field model as a probabilistic model in statistical inferences.The statistical quantities for the Gauss Markov random field model can be calculated by using the multi-dimensional Gaussian integral formulas.


Prior Probability in Bayesian Inference

( )

( )⎟⎠⎞

⎜⎝⎛ +−

+=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−−= ∑∑

∈∈

xCIxCI

xxxZ

xP

V

Vii

Ejiji

rr

r

)(21exp

)2(det

21)(

21exp

),(1,

T||

2

},{

2

αβπ

αβ

βαβα

βα

),( +∞−∞∈∀ ix ⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜

⎝

⎛

=

5

4

3

2

1

xxxxx

xr1

23

4

5{1,2}{2,3}

{3,4}{2,4}

{3,5}

11 =d 15 =d33 =d

24 =d32 =d

⎪⎩

⎪⎨

⎧∈−

==

otherwise0},{1 Eji

jidjCi

i

We adopt the Gauss Markov random field model as a prior probability of Bayesian statistics and the source signals are assumed to be generated by according to the prior probability.

I: Unit Matrix


Data Generating Process in Bayesian Statistical Inference

( ) ( )∏∈

⎟⎠⎞

⎜⎝⎛ −−=

Viii yxxyP 2

22 21exp

2

1,σπσ

σrr

1

23

4

5{1,2}{2,3}

{3,4}

{2,4}

{3,5}

1

2

3

4

5

xi yi

Additive White Gaussian Noise

As data generating processes, we assume that the observed data are generated from the source signals by adding the white Gaussian noise.


( ) ( )

yCII

I

dxdxdx,yxPxyh V

r

Lrrr

Lrr

)(

,,,,,

2

||21

αβσ

σβασβα

++=

= ∫ ∫ ∫+∞∞−

+∞∞−

+∞∞−

Bayesian Statistics

( ) ( ) ( )( )σβα

βασσβα

,,,,

,,,yP

xPxyPyxP r

rrrrr

=

xr g

( )βα ,xP r ( )σ,xyP rr yrSource Signal Data

Prior Probability Density Function

Posterior Probability Density Function

Data Generating Process


Prior ProbabilityDensityFunction

Statistical Performance by Sample Average

( )xP r

)1(xr

)2(xr

)3(xr




( )xP r

)1,1(y

)1(xr

)2(xr

)3(xr

)2,1(yr

)1,2(yr

)2,2(yr

)1,3(yr

)2,3(yrData Generating Process

( ))2(xyP r

( ))1(xyP rr

( ))3(xyP rr





( )xP r

)1,1(y

)1(xr

)2(xr

)3(xr

)2,1(yr

)1,2(yr

)2,2(yr

)1,3(yr


( ))2(xyP r

( ))1(xyP rr

( ))3(xyP rr

)1,1(hr

)2,1(hr

)1,2(hr

)2,2(hr

)1,3(hr

)2,3(hr

( ))1,1(yxP rr

( ))2,1(yxP rr

( ))1,2(yxP rr

( ))2,2(yxP rr

( ))1,3(yxP rr

( ))2,3(yxP rr





( ) ∑ ∑= =

−≡3

1

2

1

2)(),(

||61,,

k lkxlkh

VE rr

σβα

( )xP r

)1,1(y

)1(xr

)2(xr

)3(xr

)2,1(yr

)1,2(yr

)2,2(yr

)1,3(yr


( ))2(xyP r

( ))1(xyP rr

( ))3(xyP rr

)1,1(hr

)2,1(hr

)1,2(hr

)2,2(hr

)1,3(hr

)2,3(hr

( ))1,1(yxP rr

( ))2,1(yxP rr

( ))1,2(yxP rr

( ))2,2(yxP rr

( ))1,3(yxP rr

( ))2,3(yxP rr


Statistical Performance Analysis

( ) ( ) ( )

( ) ( ) ( )∫ ∫

∫ ∫

−=

−≡

ydxdxPxyPxyhV

ydxdyxPxyhV

E

rrrrrrrr

rrrrrrr

βασσβα

σβασβασβα

,,,,,1

,,,,,,1,,

2

2

( )σβα ,,,yh rr

g

( )βα ,xP r yrPrior Probability Density Function


( ),σxyP rrxr


( )σ,yxP ,,βαrr


The exact expression of the average for the mean square error with respect to the source signals and the observable data can be derived.

Statistical Performance Analysis

( ) ( ) ( ) ( )

)(Tr1

,,,,,1,,

2

2

2

CIII

V

xdydxPxyPxyhV

E

αβσσ

βασσβασβα

++=

−= ∫ ∫rrrrrrrr

( ) ( )⎟⎠⎞

⎜⎝⎛ +−

+= xCIxCIxP V

rrr )(21exp

)2(det, T

|| αβπ

αββα

( ) ( )∏∈

⎟⎠⎞

⎜⎝⎛ −−=

Viii yxxyP 2

221exp

21,

σσπσrr

( )( )( )⎪

⎩

⎪⎨

⎧∈−

==

otherwise0},{1 Eji

jidjCi

i

( )

yCII

Iyh

r

rr

)(

,,,

2 αβσ

σβα

++=


Prior Probability Density Function


Erdos and Renyi (ER) modelThe following procedures are repeated:• Choose a pair of vertices {i, j} randomly.• Add a new edge and connect to the selected

vertices if the pair of vertices have no edge.

0.0

0.1

0.2

0.3

0.4

0.5

0 5 10 15 20

P(k

)k0 5 10 15 20

0

P(d) Poisson Distribution

d

0.5

Random Network


Barabasi and Albert (BA) modelThe following procedures are repeated:• Choose a vertex i with the probability which

is proportional to the degree of vertex i.• Add a new vertex with an edge and connect

to the selected vertices.

Scale Free Network

21

21

1)2(1 =X 1)2(1 =X

2)3(1 =X 1)3(2 =X

1)3(3 =X41

41

42

10-5

10-4

10-3

10-2

10-1

100

100 101 102 103

P(k

)

k

P(d)

d


Assign a fitness parameter μ(i) to each vertex i using the uniform distribution on the interval [0, 1].

Ohkubo and Horiguchi (OH) modelThe following procedures are repeated:• Select an edge {i, j} randomly.• Select a vertex k preferentially with the

probability that is proportional to (dk + 1)μ(k)

• Rewire the edge {i, j} to {i,k} if {i,k} is not edge.

Scale Free Network

i

j

k ki

j


Statistical Performance for GMRF model on Complex Networks

Random Network by ER model

Scale Free Network by OH model

Scale Free Network by BA model

Remove all the isolated vertices


SummarySummary

Statistical Performance of Probabilistic Statistical Performance of Probabilistic Inference by Gauss Markov Random field Inference by Gauss Markov Random field models has been derived for various models has been derived for various complex networks.complex networks.We have given some numerical calculations We have given some numerical calculations of statistical performances for various of statistical performances for various complex networks including Scale Free complex networks including Scale Free Networks as well as Random Networks. Networks as well as Random Networks.

statistical inferences by gaussian markov random …kazu/tanaka-usui...10 december, 2008 cimca2008...

Documents