physics fluctuomatics (tohoku university) 1 physical fluctuomatics 2nd mathematical preparations...

Physics Fluctuomatics (Tohoku University) 1

Physical Fluctuomatics2nd Mathematical Preparations (1): Probability and statistics

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/

http://www.smapip.is.tohoku.ac.jp/~kazu/


Probability

a. Event and Probabilityb. Joint Probability and Conditional

Probabilityc. Bayes Formula, Prior Probability and

Posterior Probabilityd. Discrete Random Variable and

Probability Distributione. Continuous Random Variable and

Probability Density Functionf. Average, Variance and Covarianceg. Uniform Distributionh. Gauss Distribution


Event, Sample Space and Event

Experiment: Experiments in probability theory means that outcomes are not predictable in advance. However, while the outcome will not be known in advance, the set of all possible outcomes is knownSample Point: Each possible outcome in the experiments.

Sample Space ： The set of all the possible sample points in the experiments

Event ： Subset of the sample space

Elementary Event ： Event consisting of one sample point

Empty Event ： Event consisting of no sample point


Various Events

Whole Events Ω:Events consisting of all sample points of the sample space.Complementary Event of Event A: Ac=Ω╲ADefference of Events A and B: A╲BUnion of Events A and B: A∪BIntersection of Events A and B: A∩BEvents A and B are exclusive of each other: A∩B=ФEvents A, B and C are exclusive of each other: [A∩B=Ф]Λ[B∩C=Ф]Λ[C∩A=Ф]


Empirically Definition of Probability

Statistical Definition: Let us suppose that an event A occur r times when the same experiment are repeated R times. If the ratio r/R tends to a constant value p as the number of times of the experiments R go to infinity, we define the value p as probability of event A.

RpR

r pA Pr

Definition by Laplace: Let us suppose that the total number of all the sample points is N and they can occur equally Likely. Probability of an event A with N sample points is defined by p=n/N.


Definition of Probability

0Pr AAxion 1:

1Pr Axion 2:

BABA PrPrPr

Axion 3: For every events A, B that are exclusive of each other, it is always valid that

Definition of Kolmogorov: Probability Pr{A} for every event A in the specified sample space Ω satisfies the following three axioms:


Joint Probability and Conditional Probability

AABBA

A

BAAB

PrPr,Pr

Pr

,PrPr

A

B

Conditional Probability of Event A when Event B has happened.

Probability of Event A }Pr{AJoint Probability of Events A and B

BABA Pr,Pr


Joint Probability and Independency of Events

BAB PrPr A

B

In this case, the conditional probability can be expressed as

Events A and B are independent of each other

BABA PrPr,Pr

A

B


Marginal Probability

M

ii BAB

1

,PrPr

Let us suppose that the sample space W is expressed by Ω=A1∪A2 …∪ ∪AM where every pair of events Ai and Aj is exclusive of each other.

Marginal Probability of Event B for Joint Probability Pr{Ai,B} Marginalize

Ai B

A

BAB ,PrPrA BSimplified Notation

Summation over all the possible events in which every pair of events are exclusive of each other.


Four Dimensional Point Probability and Marginal Probability

A C D

DCBAB ,,,PrPr

Marginal Probability of Event B

A B

C D

Marginalize


Derivation of Bayes Formulas

BBABA PrPr,Pr



AABBA PrPr,Pr

BBABA PrPr,Pr



AABBA PrPr,Pr

B

BABA

Pr

,PrPr

BBABA PrPr,Pr



AABBA PrPr,Pr

B

AAB

B

BABA

Pr

PrPr

Pr

,PrPr

BBABA PrPr,Pr



AABBA PrPr,Pr

A

BA

AAB

B

AAB

B

BABA

,Pr

PrPr

Pr

PrPr

Pr

,PrPr

BBABA PrPr,Pr

A

BAB ,PrPr



AABBA PrPr,Pr

AA

AAB

AAB

BA

AAB

B

AAB

B

BABA

PrPr

PrPr

,Pr

PrPr

Pr

PrPr

Pr

,PrPr

BBABA PrPr,Pr

A

BAB ,PrPr



AABBA PrPr,Pr

AA

AAB

AAB

BA

AAB

B

AAB

B

BABA

PrPr

PrPr

,Pr

PrPr

Pr

PrPr

Pr

,PrPr

A

B

BBABA PrPr,Pr

A

BAB ,PrPr


Bayes Formula

Posterior Probability

Prior Probability

A

AAB

AABBA

PrPr

PrPrPr

A

BIt is often referred to as Bayes Rule.

Bayesian Network


Probability and Random Variable

We introduce a one to one mapping X(A) from every events A to a mutual different real number. The mapping X(A) is referred to as Random Variable of A. The random variable X(A) is often denoted by just the notation X.Probability of the event X=x that the random variable X takes a real number x is denoted by Pr{X=x}. Here, x is referred to as the state of the random variable X ． The set of all the possible states is referred to as State Space.

If events X=x and X=x’ are exclusive of each other, the states x and x’ are excusive of each other.


Discrete Random Variable and Continuous Random Variable

Discrete Random Variable: 　 Random Variable in Discrete State Space　　　　　　　　　　 Example:{x1,x2,…,xM}

Continuous Random Variable: 　 Random Variable in Continuous State Space　　　　　　　　　　 Example ： (−∞,+∞)


Discrete Random Variable and Probability Distribution

MxxxxxPxX ,,, Pr 21

If all the probabilities for the events X=x1, X=x2,…, X=xM are expressed in terms of a function P(x) as follows:

the function P(x) and the variable x are referred to as Probability Distribution and State Variable, respectively.

Random Variable State Variable State

Let us suppose that the sample W is expressed by Ω=A1∪A2 …∪ ∪AM where every pair of events Ai and Aj are exclusive of each other.We introduce a one to one mapping X:Ai xi (i=1,2,…,M).


Discrete Random Variable and Probability Distribution

MixP i ,,2,1 10

M

iixP

1

1

Probability distributions have the following properties:

Normalization Condition


Average and Variance

M

iii xPxXE

1

Average of Random Variable X : μ

M

iii xPxXV

1

22

Variance of Random Variable X: σ2

s ： Standard Deviation


Discrete Random Variable and Joint Probability Distribution

yxPyYxX ,,Pr

If the joint probability Pr{(X=x)∩(Y=y)}= Pr{X=x,Y=y} is expressed in terms of a function P(x,y) as follows:

P(x,y) is referred to as Joint Probability Distribution.

Probability Vector

Y

XState Vector

y

x


Discrete Random Variable and Marginal Probability Distribution

M

iiY yxPyP

1

,Marginal Probability Distribution

xY yxPyP ,

Summation over all the possible events in which every pair of events are exclusive of each other.

Simplified Notation

1),( x y

yxP Normalization Condition

Let us suppose that the sample W is expressed by Ω=A1∪A2 …∪∪AM where every pair of events Ai and Aj are exclusive of each other.

We introduce a one to one mapping X:Ai xi (i=1,2,…,M).


Discrete Random Variable and Marginal Probability

x z u

Y uzyxPyP ,,,

Marginal Probability Distribution

X Y

Z U

Marginalize

Marginal Probability of High Dimensional Probability Distribution


Independency of Discrete Random Variable

If random variables X and Y are independent of each others,

yPxPyxP 21,

Joint Probability Distribution of Random Variables X and Y Probability Distribution of

Random Variable X

Probability Distrubution of Random Variable Y

yPyxPyPx

Y 2, Marginal Probability Distribution of Random Varuiable Y

1)()( 21 yx

yPxP


Covariance of Discrete Random Variables

M

i

N

jjiYjXi yxPyxYX

1 1

,,Cov

Covariance of Random Variables X and Y

M

i

N

jjiiX yxPxX

1 1

,]E[

M

i

N

jjiiY yxPyY

1 1

,]E[

]V[],Cov[

],Cov[]V[

YXY

YXXR

][],Cov[ XVXX ][],Cov[ YVYY

Covariance Matrix


Example of Probability Distribution

aX tanhE

2tanh1V aX

1

cosh2

exp)( x

a

axxP

a

E[X]

0


Example of Joint Probability Distributions

a

XYYX

tanh

E],Cov[

1V X

1 ,1

cosh4

exp),( yx

a

axyyxP

a

Cov[X,Y]

0

0E X


Example of Conditional Probability Distribution

a

axyppxyP yxyx

cosh2

exp1)( ,,1

1 ,1 yx

p

pa

1ln

2

1

Conditional Probability of Binary Symmetric Channel


Continuous Random Variable and Probability Density Function

aXbXbXa PrPrPr

For a random variable X defined in the state space (−∞,+∞), the probability that the state x is in the interval (a,b) in expressed as

xXxF Pr Distribution Function

b

adxxaFbFbXa Pr

dx

xdFx Probability Density Function


Continuous Random Variable and Probability Density Function

xx 0

1

dxx

Normalization Condition


Average and Variance of Continuous Random Variable

dxxxXE

Average of Random Variable X

dxxxXV 22

Variance of Random Variable X


Continuous Random variables and Joint Probability Density Function

確率変数 X と Y の状態空間 (−∞,+∞) において状態 x と y が区間 (a,b)×(c,d) にある確率

d

c

b

adxdyyx

dYcbXa

,

Pr

Joint Probability Density Function

1, dxdyyx Normalization Condition

For random variables X and Y defined in the state space (−∞,+∞), the probability that the state vector (x,y) is in the region (a,b)(c,d) is expressed as


Continuous Random Variables and Marginal Probability Density Function

Marginal Probability Density Function of Random Variable Y

dxyxyY

,


Independency of Continuous Random Variables

Random variables X and Y are independent of each other.

yxyx 21,

Joint Probability Density Function of X and Y

Probability Density Function of Y

ydxyxyY 2,

Marginal Probability Density Function Y

1)(

1)(

2

1

dyy

dxx

Probability Density Function of X


Covariance of Continuous Random Variables

dxdyyxyxYX YX

,,Cov

Covariance of Random Variables X and Y

dxdyyxxXX

,]E[

]V[],Cov[

],Cov[]V[

YXY

YXXR

][],Cov[ XVXX ][],Cov[ YVYY

Covariance Matrix

dxdyyxyYY

,]E[


Uniform Distribution U(a,b)

xbax

bxaabx

,0

1

2

Eba

X

12

V2ab

X

Probability Density Function of Uniform Distribution

p(x)

x0 a b

(b-a)-1


Gauss Distribution N(μ,σ2)

2

22 2

1exp

2

1

xx

XE 2V X

2

2

1exp 2 d

The average and the variance are derived by means of Gauss Integral Formula

Probability Density Function of Gauss Distribution with average μ and variance σ2

xp(x)

μ x0

)0(


Multi-Dimensional Gauss Distribution

Y

XYX y

xyxyx

1

2,

2

1exp

det2

1, C

C

CC det2

2

1exp 1T dd

by using the following d -dimentional Gauss integral formula

For a positive definite real symmetric matrix C, two-Dimensional Gaussian Distribution is defined by

yx ,

C

]V[],Cov[

],Cov[]V[

YXY

YXXThe covariance matrix is given in terms of the matrix C as follows:


Law of Large Numbers

)( )(1

21 nXXXn

Y nn

Let us suppose that random variables X1,X2,...,Xn are identical and mutual independent random variables with average . m Then we have

Central Limit Theorem

)(1

21 nn XXXn

Y

tends to the Gauss distribution with average m and variance s2/n as n+.

We consider a sequence of independent, identical distributed random variables, {X1,X2,...,Xn}, with average m and variance s2. Then the distribution of the random variable


Summary


Probabilityc. Bayes Formula, Prior Probability and

Posterior Probabilityd. Discrete Random Variable and

Probability Distributione. Continuous Random Variable and


Last Talk

Present Talk


Practice 2-1

1

cosh2

exp)( x

a

axxP

Let us suppose that a random variable X takes binary values 1 and the probability distribution is given by

Derive the expression of average E[X] and variance V[X] and draw their graphs by using your personal computer.


Practice 2-2

1 ,1

cosh4

exp),( yx

a

axyyxP

Derive the expressions of Marginal Probability Destribution of X, P(X), and the covariance of X and Y, Cov[X,Y].

Let us suppose that random variables X and Y take binary values 1 and the joint probability distribution is given by


Practice 2-3

yxyx ppxyP ,, 1)( 1

p

pa

1ln

2

1 a

axyxyP

cosh2

exp)(

Show that it is rewritten as

Hint 1 ,1 12

1, yxxyyx pp lnexp

cosh(c) is an even function for any real number c.

Let us suppose that random variables X and Y take binary values 1 and the conditional probability distribution is given by


Practice 2-4

0

2

22

1exp

d

Prove the Gauss integral formula:

1

2

0 0

22

0

222

2

1explim2

2

1

2

1explim2

2

1explim2

2

1explim

2

1exp

rR

R R

R

R

R

R

RR

rdrdd

ddd

Hint


Practice 2-5

2

22 2

1exp

2

1

xxp

XE 2V X

Prove that the average E[X] and the variance V[X] are given by

Let us suppose that a continuous random variable X takes any real number and its probability density function is given by

x

Draw the graphs of p(x) for μ=0, σ=10, 20, 40 by using your personal computer.


Practice 2-6

Make a program for generating random numbers of uniform distribution U(0,1). Draw histgrams for N generated random numbers for N=10, 20, 50, 100 and 1000.

rand()randmax

1x

In the C language, you can use the function rand() that generate one of values 0,1,2,…,randmax, randomly. Here, randmax is the maximum value of outputs of rand().


Practice 2-7

Make a program that generates random numbers of Gauss distribution with average m and variance σ2. Draw histgrams for N generated random numbers for N=10, 20, 50, 100 and 1000.

For n random numbers x1,x2,…,xn generated by any probability distribution, (x1+x2+…+xn )/n tends to the Gauss distribution with average m and variance σ2/n for sufficient large n. [Central Limit Theorem]

61221 xxx

First we have to generate twelve uniform random numbers x1,x2,…,x12 in the interval [0,1].

Gauss random number with average 0 and variaince 1

σξ+μ generate Gauss random numbers with average μ and variance σ2

Hint:


Practice 2-8

CC det2

2

1exp 1T dd

For any positive integer d and d d positive definite real symmetric matrix C, prove the following d-dimensional Gauss integral formulas:

13

2

1

000

000

000

000

UUC

d

duuuU

,,, 11

By using eigenvalues λi and their corresponding eigenvectors (i=1,2,…,d) of the matrix C, we haveiu

Hint:


Practice 2-9

xxxp

d

1T

2

1exp

det2

1C

C

d

dx

x

x

x ,2

1

Prove that the average vector is and the covariance matrix is C.

We consider continuous random variables X1,X2,…,,Xd. The joint probability density function is given by


Summary


Probabilityc. Bayes Formulas, Prior Probability and

Posterior Probabilityd. Discrete Random Variable and Probability

Distributione. Continuous Random Variable and


physics fluctuomatics (tohoku university) 1 physical fluctuomatics 2nd mathematical preparations...

Documents

probability b

b events

joint probability of

b slide

prior probability

probability theory

complementary event

b union of events