hopfield rapor

Upload: samet-comert

Post on 06-Apr-2018

245 views

Category:

Documents


0 download

TRANSCRIPT

  • 8/3/2019 Hopfield rapor

    1/16

  • 8/3/2019 Hopfield rapor

    2/16

    1 INTRODUCTIONJohn J. Hopfield published two papers about an associative model that is one of the

    milestones in the field of neural networks. This neural network needs no synchronization and

    it is very successful at optimization and pattern recognition.Many of ideas in these papers were based on the previous work of other researchers.

    These researchers had considered more general associative memory models in the 1970s, but

    by restricting the architecture of the network to a symmetric connection matrix with a zero

    diagonal, it was possible to design recurrent networks with stable states.

    In addition to this there are several other reasons why Hopfields papers have had such

    an impact. First, he identified a close analogy between his neural network and the Ising model

    of magnetic material. Hopfield also had close contacts with VLSI chip designers, because of

    the bell laboratories. The fact that Hopfield addressed the implementation issues of his

    network distinguished him from most previous neural network researchers.

    1.1 Terminology:y Symmetric connections: if the weight between neurons doesnt change it is asymmetric connection.

    y Asymmetric connections: if the weight between neurons doesnt change it is asymmetric connection.

    Symmetric and Asymmetric Connections

    y Asynchronous: when firing a neuron, first calculates the weighted input sum andupdates immediately. This can be done in a fixed order, or neurons can be picked at random,

    which is called asynchronous random updating.y Synchronous: the weighted input sums of all neurons are calculated without updatingthe neurons. Then all neurons are set to their new value, according to the value of their

    weighted input sum.

    y Associative memory: Recalling a previously experienced item by thinking ofsomething that is linked with it, thus invoking the association. For example when we see red

  • 8/3/2019 Hopfield rapor

    3/16

    light we understand it as stop and green as go. So colors are reminds us the commands.

    This is a associative memory.

    y Ising model: this physical system consists of particles that capable of adopting one oftwo states. In the case of ferromagnetic materials, their atoms can be modeled as particles of

    spin 1/2 (up) or spin 1/2 (down). The spin points in the direction of the magnetic field. All

    tiny magnets interact with each other. This causes some of the atoms to flip their spin until

    equilibrium is reached and the total magnetization of the material reaches a constant level,

    which is the sum of the individual spins.

    Ising model

    2 HOPFELD NEURAL NETWROK2.1 A physical analogy with memory

    To show the similarity between the energy and the memory, we can think a bowl in

    which a ball bearing is allowed to roll freely as shown in figure below. Suppose we let the ball

    go from a point somewhere up the side of the bowl. The ball will oscillates around bottom of

    the bowl and after some oscillations ball will be rest at the bottom. We can describe what

    happened, in terms of energy. At first, system have a potential energy according to its height,

    higher the ball the greater its energy.

  • 8/3/2019 Hopfield rapor

    4/16

    Eventually the ball comes to rest at the bottom, energy is now at a minimum since any

    other location of the ball is associated with some potential energy, which may be lost on

    allowing the bowl to reach equilibrium. This equilibrium state doesnt change with the

    starting point. There is another way of thinking about this process that ties in with our ideas

    about memory. This example illustrated to understand the relation. Thus, we suppose that the

    ball comes to rest in the same place each time because it remembers where the bottom of

    the bowl is.

    If we now use a corrugated surface instead of a single depression we may store many

    memories. If the ball is now started somewhere on this surface, it will eventually come to

    rest at the local depression that is closest to its initial starting point. That is, it evokes the

    stored pattern which is closest to its initial partial pattern or cue. This corresponds to an

    energy minimum of the system. The memories shown correspond to states x1, x2, x3 where

    each of these is a vector.

    There are therefore two complementary ways of looking at what is happening. One is to

    say that the system falls into an energy minimum; the other is that it stores a set of patterns

    and recalls that which is closest to its initial state.

    2.2 Hopfield networkNow we are going to apply principle we talk above, to the construction of a neural

    network capable of performing associative recall. Consider the network consisting of three

    nodes shown below. Each node is connected to every other node and the connection weights

    are symmetric. Thresholds are all assumed to be zero. Notice that the flow of information in

    this type of net is not in a single direction, since it is possible for signals to flow from a node

    back to itself via other nodes. We say there is feedback in the network or that it is recurrent

    because nodes may be used repeatedly to process information.

  • 8/3/2019 Hopfield rapor

    5/16

    The state of the network at any time is given as (x1, x2, x3). Suppose we now start this

    net in some initial state, choose a node at random and let it update its output or fire. That is,

    the chosen node evaluates its activation in the normal way and outputs a 1 if this is greater

    than or equal to zero and a 0 otherwise. The net now finds itself either in the same state as it

    started in, or it changes one of the states. We now choose another node at random, let itupdate or fire, and repeat this many times. This process defines the dynamics of the net.

    In describing these state transitions it is convenient to attach a numeric label to each

    state and the most natural way of doing this is to interpret the Boolean state vector as a binary

    number, so that state (x1, x2, x3) is labeled with 4x1+2x2+x3. For example, (1, 1, 0) is state 6.

    For each network state, there are three possible outcomes for the next state depending on

    which of the three nodes is chosen to fire. Suppose, for example, the net starts in state (1, 0, 1)

    (label 5) and node 1 fires. The activation of this node is given by

    a=w13x3+w12x2=21+10=2.Then, since this is less than 0, the new output is also 0 and the new state is (0, 0, 1)

    (label 1); in summary, state 5 goes to state 1 when node 1 fires. Repeating this working for

    nodes 2 and 3 firing, the new states are 7 and 4 respectively. By working through all initial

    states and node selections it is possible to evaluate every state transition of the net as shown

    as;

    Initial State After A Node Firingx1 x2 x3 Label # 1 2 3

    0 0 0 0 4 2 1

    0 0 1 1 1 3 1

    0 1 0 2 6 2 3

    0 1 1 3 3 3 3

    1 0 0 4 4 6 4

    1 0 1 5 1 7 4

    1 1 0 6 6 6 6

    1 1 1 7 3 7 6

  • 8/3/2019 Hopfield rapor

    6/16

    Notice that a state may or may not change when a node fires. This information may also

    be represented in graphical form as a state transition diagram, shown in figure below;

    For example, starting in state 5 we see from the diagram that there is an equal

    probability of 1/3 of going to states 1, 7, or 4, which is reflected in the three arcs emanating

    from state 5 in the diagram. Again, starting in state 1 and updating nodes 1 or 3 results in no

    state change, so there is a probability of 2/3 that state 1 stays as it is; however, choosing node

    2 to update results in a transition to state 3. The no-change condition gives an arc that starts

    and ends at the same state.

    Another firing method is that; classify the units of a network according to their state: the

    first set contains the units with state 1, the second set the units with state 1. There are edges

    linking every unit with all the others, so that some edges go from one set to the other. We now

    randomly select one of the units and compute its attraction by the units in its own set and

    the attraction by the units in the other set. The attraction is the sum of the weights of all

    edges between a unit and the units in its set or in the other one. If the attraction from the

    outside is greater than the attraction from its own set, the unit changes sides by altering its

    state. If the external attraction is lower than the internal, the unit keeps its current state.[74]

  • 8/3/2019 Hopfield rapor

    7/16

    2.3 Defining an energy for the networkThe dynamics of the net are described completely by the state transition table or

    diagram. It has demonstrated the existence of stable states. On the other hand energy basedform is more accurate to find stable states. We can think every node effect other with the

    associated weight. The use of the energy concept in this context may be introduced using an

    analogy with a simple physical system. Thus, consider a pair of objects joined by a spring, use

    of a tension spring, which tends to pull the objects together. Since work has to be done to

    draw the objects apart, the energy minimum of this system occurs when the objects are close

    to each other or, in other words, tend towards the same position in space. Compression spring

    energy minimum occurs when the objects are far apart.

    Now consider two nodes i, j in a Hopfield net connected by a positive weight +w, as

    shown above. We claim that positive and negative weights are analogous to the tension and

    compression springs respectively, since positive weights tend to make the nodes take on the

    same output values while negative weights tend to force different values.

    The constraint description may now be set within the energy framework by assigning

    high energies to states that tend to get destabilized and low energies to those that are

    reinforced. One way of doing this is to define the internode energy eij by The energy of the whole network E is found by summing over all pairs of nodes

  • 8/3/2019 Hopfield rapor

    8/16

    Since the connections are symmetric, the sum includes each pair twice so we put the

    coefficient. When we adding the threshold value for every node we obtain the energy formula

    for Hopfield neural network:

    2.4 Finding the weightsp to now nothing has been said about how the network can be made to store patterns

    from a training set. In his original paper, Hopfield (1982) did not give a method for training

    the networks under an incremental, iterative process. But commonly to store pattern into the

    network biologically inspired Hebb training method is used.

    The main purpose of the Hopfield neural network is constraint the outputs of the

    network according to given weights. With this aspect, we can adjust our weights with therelations between them. For example, two nodes that tend, on average, to take on the same

    value over the training set so that the pairs 0, 0 or 1, 1 dominates; we say that the nodes are

    correlated. The pair 1, 1 will be reinforced by there being a positive internode weight. If, on

    the other hand, the two nodes tend, on average, to take on opposite values with pairs 0, 1 or 1,

    0, then this will be reinforced by a negative internode weight.

    2.4.1 The Hebb ruleIn 1949, neuropsychologist D. O. Hebb describes a possible way for biological neurons

    learn. And he said that:

    When an axon of cell A is near enough to excite a cell B and repeatedly or persistently

    takes part in firing it, some growth process or metabolic change takes place in one or

    both cells such that As efficiency, as one of the cells firing B, is increased.

    This description can be adapted to our training algorithm in three steps:

    1. Choose a pattern from the training set at random.2. Present the components of this pattern at the outputs of the corresponding nodes of the

    net.

    3. If two nodes have the same value then makes positive increment to the internodeweight. If they have opposite values then make negative decrement to the weight.

    Iteration of these three steps many times constitutes a training algorithm whose learning rule

    may be written mathematically as;

  • 8/3/2019 Hopfield rapor

    9/16

  • 8/3/2019 Hopfield rapor

    10/16

    This expression can also be written as

    States are binary so xi = xi

    2;

    Our Hopfield energy function is;

    And we can make some arrangements to our energy function to assimilate it to Hopfield

    energy function;

    For every column we find the weight as a constant -2 and threshold value -1.

    The minimum of the function corresponds to the situation in which just one rook has

    been positioned in every column. Similarly, for the rows of the board we define the function

    E2 according to

  • 8/3/2019 Hopfield rapor

    11/16

  • 8/3/2019 Hopfield rapor

    12/16

    4 APPLICATIONS OF HOPFIELD NETWORKSAs I mentioned before, when Hopfield was working for Bell industries, he proposed an

    electrical realization of his model. He used amplifiers for Hopfield units and obtains

    complements of these units with inverters and fed back these to the inputs. Symbolic

    representation of this circuit is shown in figure. In this figure circles in contacts, are

    represented resistance and Hopfield adjust the weights with these resistance.

  • 8/3/2019 Hopfield rapor

    13/16

    For example, contains a resistor with resistance r13 = 1/w13. The constants wij

    represent the weights of the Hopfield network between the units i and j. Inhibiting

    connections between one unit and another (that is, connections with negative weights) are

    simulated by connecting one inverted output of a unit to the other one.

    One of the most important implementation of Hopfield network is realized by optical

    way. The networks binary values are projected through the vertical lens to the left of the

    arrangement. The lens projects each value xi onto the corresponding row of an optical mask.

    Each row i in the mask is divided into fields which represent the n weights wi1, wi2 . . . win. on

    the mask, reflected light emitting or reflecting according to attended weight. The individual

    unit states are projected using light emitting diodes. The light going through the mask is

    collected by another lens and obtained a single value. The amount of light that goes through

    the mask is proportional to the product of x i and wij at each position ij of the mask.

    In literature Hopfield neural network used for two purposes one of them is pattern

    recognition and the other is optimization problems. In pattern recognition, binary images are

    sampled and for each field a node will be attend. When image is stored to the network weights

    will be updated. For more patterns node number will be increase exponentially.

    For optimization problems can be written in an analytical form isomorphic to the

    Hopfield energy function, it can be solved by a Hopfield network. We can assume that every

    unit in the network is simulated by a small processor. The states of the units can be computed

    asynchronously by transmitting the current unit states from processor to processor. There is

    no need for expensive synchronization and the task is solved by a massively parallel system.

    This strategy can be applied to all those combinatorial problems for whose solution large

    mainframes have traditionally been used.

  • 8/3/2019 Hopfield rapor

    14/16

    4.1 HNN in image processingBecause of the usage of associative memory Hopfield neural network (HNN) finds

    largely working area in image processing. In many studies Hopfield NN was used for image

    recognition such as hand writing recognition, face and gender recognition with genetic

    algorithm, edge detection, image segmentation, image restoration and image compression.

    In one study different characters are stored in HNN and when input is degraded and

    noise image HNN is enhance the image(it recognize the original image from the degraded

    image). Results are very successful with less recognized characters and under a fifty persent

    noise.

    As seen in the results HNN is very successful in image recognition. But we shouldnt

    forget only binary images can be an input for Hopfield neural network.

    4.2 Optimization problemsAs I mentioned before, if we can be write the given problem in an analytical form

    isomorphic to the Hopfield energy function, it can be solved by a Hopfield network. Hopfield

    and Tank were the first to try to solve the TSP using a neural network. But such a complex

    problem solved by HNN with binary neurons it strict a local minimum. So in 1984 Hopfield

    replaced binary neurons by neurons whose instantaneous activity increases with input and

    applied this to the TSP.

  • 8/3/2019 Hopfield rapor

    15/16

    Results of the travelling sales man problem are shown above. In these figures first and

    second are continuous Hopfield model and the last one is the binary Hopfield model. As its

    clearly shown in the figure continuous HNN finds the global minimum but the binary HNN

    finds the local minimum.

    Heuristics have been proposed for the TSP, starting with the classical work by

    Kernighan and Lin. One of the first to deal with limits of the Hopfield model with constant

    size for the solution of the TSP was Abu-Mostafa. Although this almost meant the death of

    the traveling salesman, the Hopfield model and its stochastic variants have been applied in

    many other fields, such as psychology, simulation of ensembles of biological neural networks,

    and chaotic behavior of neural circuits.

    5 CONCLUSIONHopfield NN is one of the most important milestones of recurrent networks. Some

    properties of this network is:

    A recurrent network with all nodes connected to all other nodes Nodes have binary outputs (either 0,1 or -1,1) Weights between the nodes are symmetric wij = wji No connection from a node to itself is allowed Nodes are updated asynchronously (i.e. nodes are selected at random) The network has no hidden nodes or layer

    Main advantages of this system are, do not need any kind of synchronization; they

    guarantee that a local minimum of the energy function will be reached. In pattern recognition

    HNN is very successful but with the increasing of the stored pattern number the network

    expands and the success of the network is decrease.

  • 8/3/2019 Hopfield rapor

    16/16

    In simple optimization problems HNN with binary neurons can be used but it only finds

    local minimum. For more complex optimization problems like travelling salesman or rout

    planning we must use HNN with continuous neuron.

    There are a lot of areas that Hopfield used with different algorithms. But I think

    Hopfield can be very successful in classification studies. The studies in this task are much

    related and can be developed.

    1) Hopfield J., 1982, Neural networks and physical systems with emergent collectivecomputational abilities, Proc. Natl. Acad. Sci. USA

    2) Rojas, R., 1996, Neural Networks A System Introduction, Springer-Verlag Berlin3) Gurney, K., 1997, An Introduction to Neural Networks ,UCL Press, London4) Hopfield J., 1985, Neural Computation of Decisions in Optimization Problems,

    Biological Cybernetics, Springer-Verlag

    5) Boden M., 2001, AGuide to Recent Neural Networks and Backpropagation ,Halmstad University