# chapter 5 recurrent networks and temporal feedforward networks

Post on 12-Jan-2016

51 views

Embed Size (px)

DESCRIPTION

Chapter 5 Recurrent Networks and Temporal Feedforward Networks. (Chuan-Yu Chang ) Office: ES 709 TEL: 05-5342601 ext. 4337 E-mail: chuanyu@yuntech.edu.tw. Overview of Recurrent Neural Networks. - PowerPoint PPT PresentationTRANSCRIPT

Chapter 5Recurrent Networks and Temporal Feedforward Networks (Chuan-Yu Chang ) Office: ES 709TEL: 05-5342601 ext. 4337E-mail: chuanyu@yuntech.edu.tw

Chuan-Yu Chang Ph.D.*

Overview of Recurrent Neural NetworksA network that has closed loops in its topological structure is considered a recurrent network.Feedforward networks:Implemented fixed-weighted mapping from input space to output space.The state of any neuron is solely determined by the input to the unit and not the initial and past states of the neuron.Recurrent neural networksRecurrent neural networks utilize feedback to allow initial and past state involvement along with serial processing.Fault-tolerantThese networks can be fully connected.The connection weights in a recurrent neural network can be symmetric or asymmetric.In symmetric case, (wij=wji) the network always converges to stable point. However, these networks cannot accommodate temporal sequences of pattern.In the asymmetric case, (wijwji) the dynamics of the network can exhibit limit cycles and chaos, and with the proper selection of weights, temporal spatial patterns can be generated and stored in the network.

Chuan-Yu Chang Ph.D.*

Hopfield Associative MemoryHopfield(1988)The physical systems consisting of a large number of simple neurons can exhibit collective emergent properties.A collective property of a system cannot emerge from a single neuron, but it can emerge from local neuron interactions in the system.Produce a content-addressable memory that can correctly yield an entire memory from partial information.

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)The standard discrete-times Hopfield neural networkA kind of recurrent networkCan be viewed as a nonlinear associative memory, or content-addressable memory.To perform a dynamic mapping function.Intended to perform the function of data storage and retrieval.The network stores the information in a dynamically stable environment.A stored pattern in memory is to be retrieved in response to an input pattern that is a noisy version (incomplete) of the stored pattern.

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)Content-addressable memory (CAM)Attractor is a state that the system will evolve toward in time, starting from a set of initial conditions. (basin of attraction)If an attractor is a unique point in the state space, it is called a fixed point.A prototype state Fh is represented by a fixed point sh of the dynamic system.Thus, Fh is mapped onto the stable points sh of the network.

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)Activation function: symmetric hard-limiterOutput can only be +1 or -1The output of a neuron is not fed back to itself. Therefore, Wij=0 for i=j.

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)The output of the linear combiner is written as where is the state of the networkThe state of each neuron is given by if vi=0, the value of xj will be defined as its previous state.The vector-matrix form of (5.1) is given byExternal threshold(5.1)(5.2)(5.3)

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)The network weight matrix W is written as Each row in (5.4) is the associated weight vector for each neuron.The output of the network can be written as vector-matrix form Scalar form (5.4)(5.5)(5.6)

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)There are two basic operational phases associated with the Hopfield network: the storage phase and the recall phase.During the storage phase, the associative memory is build according to the outer-product rule for correlation matrix memories.Given the set of r prototype memories, the network weight matrix is computed as

Recall phaseA test input vector xThe state of network x(k) is initialized with the values of the unknown input, ie., x(0)=x.Using the Eq.(5.6), the elements of the state vector x(k) are updated one at a time until there is no significant change in the elements of the vector. When this condition is reached, the stable state xe is the network output.

(5.7)wij=0 for i=j

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)Discrete-time Hopfield network training algorithmStep 1: (storage phase) Given a set of prototype memories, using (5.7), the synaptic weights of the network are calculated according to

Step 2: (Recall Phase) Given an unknown input vector x, the Hopfield network is initialized by setting the state of the network x(k) at time k=0 to x Step 3: The element of the state of the network x(k) are update asynchronously according to (5.6) This iterative process is continued until it can be shown that the element of the state vector do not change. When this condition is met, the network outputs the equilibrium state

(5.8)(5.9)(5.10)(5.11)

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)The major problem associated with the Hopfield network is spurious equilibrium state.These are stable equilibrium states that are not part of the design set of prototype memories.spurious equilibrium stateThey can result from linear combinations of an odd number of patterns.For a large number of prototype memories to be stored, there can exist local minima in the energy landscape.Spurious attractors can result from the symmetric energy function.Li et al., proposed a design approach, which is based on a system of first-order linear ordinary differential equation.The number of spurious attractors is minimized.

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)Because the Hopfield network has symmetric weights and no neuron self-loop, an energy function (Lyapunov function) can be defined.An energy function for the discrete-time Hopfield neural network can be written as

The change in energy function is given by(5.12)(5.13)The operation of Hopfield network leads to a monotonically decreasing energy function, and changes in the state of the network will continue until a local minimum of the energy landscape is reached/x is the state of the network,x is an externally applied input presented to the networkq is the threshold vector.

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)For no externally applied inputs, the energy function is given by

The energy change is

The storage capacity (bipolar patterns) of the Hopfield network is approximately by

If most of the prototype memories be recalled perfectly, the maximum storage capacity of the network given by

If it is required that 99% of the prototype memories are to be recalled perfectly (5.14)(5.15)(5.16)(5.17)(5.18)n is the number of neurons in the network

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)Example 5.1threshold0(bipolar vector)(prototype memory) [-1, 1, -1][1, -1, 1](5.7)

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.) [-1, -1, 1], [1, -1, -1] [1, 1, 1][ 1, -1, 1](5.14) 5.1,energy

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)Example 5.212*12(+1-1)12*12=144144*144=20736Threshold q=0(-1+1)

Chuan-Yu Chang Ph.D.*

Hopfield Associative Memory (cont.)5prototype vector(5.7)5.730% bit error rate0

Chuan-Yu Chang Ph.D.*

The Traveling-Salesperson ProblemOptimization problemsFinding the best way to do something subject to certain constraints.The best solution is defined by a specific criterion.In many cases optimization problems are described in terms of a cost function.The Traveling-Salesperson Problem, TSPA salesperson must make a circuit through a certain number of cities.Visiting each city only once.The salesperson returns to the starting point at the end of the trip.Minimizing the total distance traveled.

Chuan-Yu Chang Ph.D.*

The Traveling-Salesperson Problem (cont.)ConstraintWeak constraintEg. Minimum distanceStrong constraintConstraints that must be satisfied.The Hopfield network is guaranteed to converge to a local minimum of the energy function.To use the Hopfield memory for optimization problems, we must find a way to map the problem onto the network architecture.The first step is to develop a representation of the problems solutions that fit an architecture having a single array of PEs.

Chuan-Yu Chang Ph.D.*

The Traveling-Salesperson Problem (cont.)Hopfield(Lyapunov energy function)

Wq

Chuan-Yu Chang Ph.D.*

The Traveling-Salesperson Problem (cont.)An energy function must satisfies the following criteria.Visit each city only once on the tourVisit each position on the tour in a timeInclude all n cities.The shortest total distances.

Chuan-Yu Chang Ph.D.*

The Traveling-Salesperson Problem (cont.)The energy equation isN

Chuan-Yu Chang Ph.D.*

The Traveling-Salesperson Problem (cont.)Comparing the cost function and the Lyapunov function of the Hopfield networks, the synaptic interconnection strengths and the bias input of the network are obtained as where the Kronecker delta function defined as .

Chuan-Yu Chang Ph.D.*

The Traveling-Salesperson Problem (cont.)The total input to neuron (x,i) is

A Contextual Hopfield Neural Networks for Medical Image Edge Detection (Chuan-Yu Chang)Optical Engineering, vol. 45, No. 3, pp. 037006-1~037006-9,2006. (EISCI)

Chuan-Yu Chang Ph.D.*

IntroductionEdge detection from medical images(such as CT and MRI) is an important steps in the medical image understanding system.

Chuan-Yu Chang Ph.D.*

IntroductionThe Proposed CHNNThe input of CHNN is the original two-dimensional image and the output is an edge-based feature map. Taking each pixels contextual information.Experimental results are more perceptual than the CHEFNN.The execution time is fast than

Recommended