establishing the equivalence between recurrent neural networks and turing machines. ritesh kumar...

Establishing the Equivalence between Recurrent Neural Networks and Turing Machines.

Ritesh Kumar Sinha(02d05005)

Kumar Gaurav Bijay(02005013)

“No computer has ever been designed that is everaware of what it’s doing; but most of the time, wearen’t either”

– Marvin Minsky.

Introduction

Plan : Establish Equivalence between recurrent neural network and turing machine History of Neurons Definitions Constructive Proof of equivalence

Approach : Conceptual Understanding

Motivation

Understanding the learning patterns of human brain – concept of neuron

Is Turing Machine the ultimate computing machine ?

How powerful are neural networks – DFA, PDA, Turing Machine, still higher …

Brain

The most complicated human organ sense, perceive, feel, think, believe, remember, utter Information processing Centre

Neurons : information processing units

MCP Neuron

McCulloch and Pitts gave a model of a neuron in 1943

But it's only a highly simplified model of real neuron Positive weights (activators) Negative weights (inhibitors)

Artificial Neural Neworks (ANN)

Interconnected units : model neurons

Modifiable weights (models synapse)

Types of ANN

Feed-forward Networks Signals travel in one way only Good at computing static functions

Neuro-Fuzzy Networks Combines advantages of both fuzzy reasoning

and Neural Networks. Good at modeling real life data.

Recurrent Networks

Recurrent Neural Networks

Activation Function:f(x) = x , if x > 0 0 , otherwise

Turing Machine and Turing Complete Languages

Turing Machine: As powerful as any other computer

Turing Complete Language: Programming language that can compute any

function that a Turing Machine can compute.

A language L

Four basic operations: No operation : V V Increment : V V+1 Decrement : V max(0,V-1) Conditional Branch : IF V != 0 GOTO j

(V is any variable having positive integer values, and j stands for line numbers)

L is Turing Complete

Turing Machine can encode Neural Network

Turing machine can compute any computable function, by definition

The activation function, in our case, is a simple non-linear function

Turing machine can therefore simulate our recurrent neural net

Intuitive, cant we write code to simulate our neural net

An example

Function that initiates: Y=X

L1: X X - 1

Y Y + 1

if X != 0 goto L1

An example

Function that initiates: Y=X

if X != 0 goto L1

X X + 1

if X != 0 goto L2

L1: X X - 1

Y Y + 1

if X != 0 goto L1

L2: Y Y

L is Turing Complete : Conceptual Understanding

Idea: Don’t think of C++, think of 8085 ;)

Subtraction: Y = X1 - X2



Subtraction: Y = X1 - X2 Yes, decrement X, increment Y, when Y=0,

stop

Multiplication, division:



Subtraction: Y = X1 - X2

Yes, decrement X, increment Y, when Y=0, stop

Multiplication, division:

Yes, think of the various algos you studied in Hardware Class :)


If: if X=0 goto L


If: if X=0 goto L

Yes,

if X != 0 goto L2

Z Z + 1 // Z is dummy

if Z != 0 goto L

L2: ...

Constructing a Perceptron Network for Language L

For each variable V : entry node NV

For each program row i : instruction node Ni

Conditional branch instruction on row i :

2 transition nodes Ni’ and Ni”

Constructions

Variable V :

NV NV

No Operation:

Ni Ni+1

Constructions Continued

Increment Operation :

Ni Ni+1

Ni NV

Decrement Operation:

Ni Ni+1

Ni NV

Constructions Continued

Conditional Branch Operation :

Ni Ni’

Ni Ni’’

Nv Ni’’

Ni’’ Ni+1

Ni’ Nj

Ni’’ Nj

Definitions

Legal State: Transition Nodes (Ni’ and Ni’’) : 0 outputs

Exactly one Ni has output=1

Final State : All instruction nodes have 0 outputs.

Properties

If yi = 1 then program counter is on line i

V = output of Nv

Changes in network state are activated by non-zero nodes.

Proof

•If row i is V V

•If row i is V V - 1

•For V V + 1, behavior is similar

Proof Continued

•If row i is : if V != 0 GOTO j

Proof Continued

•Thus a legal state leads to another legal state.

What about other activation functions?

• Feed-forward Net With Binary State Function is DFA

• Recurrent Neural Network with:1) Sigmoid is Turing Universal2) Saturated Linear Function is

Turing Universal

Conclusion

Turing machine can encode Recurrent Neural Network.

Recurrent neural net can simulate a Turing complete language.

So, Turing machines are Recurrent Neural Networks !

Project

Implementation of Push down automata (PDA) using Neural Networks.

References

[1] McCulloch , W. S., and Pitts, W. : A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 1943 :115-133.

[2] Hyotyniemi, H : Turing Machines are Recurrent Neural Networks. In SYmposium of Artificial Networks, Vaasa, Finland, Aug 19-23, 1926, pp. 13-24

[3] Davis, Martin D.Weyuker, Elaine J. : Computability, complexity, and languages : fundamentals of theoretical computer science, New York : Academic Press, 1983.

References Continued

[4] J. E. Hopcroft and J. D. Ullman, Introduction to Automata Theory, Languages and Computation. Addison-Wesley, 1979.

[5] Arbib, M. : Turing Machnies, Finite Automata and Neural Nets. Journal of the ACM (JACM), NY, USA, Oct 1961, Vol 8 , Issue 4 , pp. 467 - 475.

establishing the equivalence between recurrent neural networks and turing machines. ritesh kumar...

Documents