physical fluctuomatics 1st review of probabilistic ... · 1st review of probabilistic information...

Physical Fuctuomatics (Tohoku University) 1

Physical Fluctuomatics 1st Review of probabilistic information processing

Kazuyuki Tanaka Graduate School of Information Sciences

[email protected] http://www.smapip.is.tohoku.ac.jp/~kazu/

Webpage: http://www.smapip.is.tohoku.ac.jp/~kazu/PhysicalFluctuomatics/


Textbooks

  Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) .

  Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., 2009 (in Japanese).


References of the present lecture   K. Tanaka: Statistical-mechanical approach to image

processing (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002.

  Y. Kabashima and D. Saad: Statistical mechanics of　low-density parity-check codes (Topical Review), J. Phys. A, vol.37, no.6, pp.R1-R43, 2004. H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. M. Opper and D. Saad D (eds): Advanced Mean Field Methods --- Theory and Practice, MIT Press, 2001. C. M. Bishop: Pattern Recognition and Machine Learning, Springer, 2006. M. J. Wainwright and M. I. Jordan: Graphical Models, Exponential Families, and Variational Inference, now Publishing Inc, 2008. M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2009.


Benefit of Information & Communications Technology

 Ubiquitous Computing  Ubiquitous Internet

Benefit of Information & Communications Technology

Demand for Intelligence It cannot be satisfied only with it being only cheap and being quick.


Field of Information Processing

Information processing according to theories  Inference from propositions

Realization by progress of computational processing capacity

Information processing in real world  Diversity of reason in phenomenon  Compete data is not necessarily obtained.  It is difficult to extract and select some important

information from a lot of data. Uncertainty caused by the gap of knowing simply and understanding actually. We hope to deal successfully with such uncertainty.

Information processing for numerical calculations  Definite Procedure has been given for each calculation.


Computer for next generations

Required Capacity Capability to sympathize with a user （Knowledge) Capability to put failure and experience to account in the next chance （Learning）

How should we deal successfully with the uncertainty caused by the gap of knowing simply and understanding actually?

Formulation of knowledge and uncertainty

Realization of information processing data with uncertainty


Computational model for information processing in data with uncertainty

Probabilistic Inference Probabilistic model

with graphical structure （Bayesian network） Medical diagnosis

Failure diagnosis Risk Management

Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness.

Inference system for data with uncertainty

modeling

Node is random variable. Arrow is conditional probability.

Mathematical expression of uncertainty =>Probability and Statistics

Graph with cycles

Important aspect


Computational Model for Probabilistic Information Processing

Probabilistic Information Processing Probabilistic Model

Bayes Formula

Algorithm

Monte Carlo Method  Markov Chain Monte Carlo Method  Randomized Algorithm  Genetic Algorithm

Approximate Method  Belief Propagation  Mean Field Method

Randomness and Approximation


Probabilistic Image Processing

Noise Reduction by Probabilistic Image

Processing

K. Tanaka: J. Phys. A, vol.35, 2002. A. S. Willsky: Proceedings of IEEE, vol.90, 2002.

173110218100120219202190202192

Average =⎪⎭

⎪⎬

⎫

⎪⎩

⎪⎨

⎧

192 202 190

202 219 120

100 218 110

192 202 190

202 173 120

100 218 110

Modeling of Probabilistic Image Processing based on Conventional Filters

Markov Random Field Model Probabilistic Image Processing

The elements of such a digital array are called pixels. At each point, the intensity of light is represented as an integer number or a real number in the digital image data.

Algorithm

Conventional Filter


Probabilistic Image Processing

Degraded Image (Gaussian Noise）  Probabilistic Image Processing

Lowpass Filter Wiener Filter Median Filter

MSE:520 MSE: 2137

MSE:860 MSE:767 MSE:1040

K. Tanaka: J. Phys. A, vol.35, 2002. A. S. Willsky: Proceedings of IEEE, vol.90, 2002.


Error Correcting Code

Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004.

High Performance Decoding Algorithm

010 000001111100000 001001011100001

0 1 0

code

010

error

decode

Parity Check Code

Turbo Code, Low Density Parity Check (LDPC) Code

majority rule Error Correcting Codes

14 January, 2010 Hokkaido University GCOE Tutorial

(Sapporo） 12

Error Correcting Codes and Belief Propagation

)2 (mod )2 (mod )2 (mod

5329

6438

4217

XXXXXXXXXXXX

++=

++=

++=

⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

=

⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

110011

6

5

4

3

2

1

xxxxxx

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

=

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

010110011

9

8

7

6

5

4

3

2

1

xxxxxxxxx

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

=

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

011110111

9

8

7

6

5

4

3

2

1

yyyyyyyyy

0)2 (mod 1011)2 (mod 1000)2 (mod 011

9

8

7

=++=

=++=

=++=

XXX

1 1

0

p−1

1

0

0

p−1

p

p

Received Word

Code Word

Binary Symmetric Channel

14 January, 2010 Hokkaido University GCOE Tutorial

(Sapporo） 13

Error Correcting Codes and Belief Propagation

)2 (mod )2 (mod )2 (mod

5329

6438

4217

XXXXXXXXXXXX

++=

++=

++=

Fundamental Concept for Turbo Codes and LDPC Codes


CDMA Multiuser Detectors in Mobile Phone Communication

Relationship between mobile phone communication and spin glass theory

T. Tanaka, IEEE Trans. on Information Theory, vol.48, 2002

Signals of User A

Spreading Code Sequence

Wireless Communication

Received Data

×× Decode

Spreading Code Sequence

Probabilistic model for decoding can be expressed in terms of a physical model for spin glass phenomena

Noise

Coded Signals of Other Users

Coded Signals of User A


Artificial Intelligence

Bayesian Network

CA

SA RA

WA

Probabilistic inference system

Practical algorithms by means of belief

propagation

J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).


Main Interests Information Processing:

Data Physics:

Material, Natural Phenomena

System of a lot of elements with mutual relation Common Concept between Information Sciences and Physics

Material Molecule

Materials are constructed from a lot of molecules.

Molecules have interactions of each other.

０,1 101101110001

010011101110101000111110000110000101000000111010101110101010Bit

Data

Data is constructed from many bits

A sequence is formed by deciding the arrangement of bits.

A lot of elements have mutual relation of each other Some physical concepts in Physical models are useful for the design of computational models in probabilistic information processing.


Horizon of Computation in Probabilistic Information Processing

Compensation of expressing uncertainty using probability and statistics

It must be calculated by taking account of both events with high probability and events with low probability.

Computational Complexity It is expected to break throw the computational complexity by introducing approximation algorithms.


What is an important point in computational complexity?

How should we treat the calculation of the summation over 2N configuration?

( )∑ ∑ ∑= = =FT, FT, FT,

211 2

,,,x x x

NN

xxxf !!( )

}}

} ;,,,

F){or Tfor(

F){or Tfor( F){or Tfor(

;0

21

2

1

!

"

!

L

N

xxxfaax

xx

a

+←

=

=

=

←

N fold loops

If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.


Why is a physical viewpoint effective in probabilistic information processing?

Matrials are constructed from a lot of molecules. (1023 molecules exist in 1 mol.)

Molecules have intermolecular forces of each other ( )∑∑ ∑

1 2

,,, 21x x x

NN

xxxf !!

Theoretical physicists always have to treat such multiple summation.

Development of Approximate Methods Probabilistic information processing is also usually reduced to multiple summations or integrations.

Application of physical approximate methods to probabilistic information processing


Academic Circulation

Academic Circulation

Academic Circulation between Physics and Information Sciences

Physics Information Sciences

Understanding and prediction of properties of materials and natural phenomena

Extraction and processing of information in data

Common Concept Statistical Mechanical Informatics

Probabilistic Information Processing

Statistical Sciences


Summary of the present lecture

 Probabilistic information processing  Examples of probabilistic information processing  Common concept in physics and information sciences  Application of physical modeling and approximations

Future Lectures

 Fundamental theory of probability and statistics  Linear model

 Graphical model . . .

physical fluctuomatics 1st review of probabilistic ... · 1st review of probabilistic information...

Documents