ARTIFICIAL NEURAL NETWORKS Sai Anjaneya

WHAT ARE NEURAL NETWORKS?• In machine learning and cognitive

science, artificial neural networks (ANNs) are a family of models inspired by biological neural networks (the central nervous systems of animals, in particular the brain) and are used to estimate or approximate functions that can depend on a large number of inputs and are generally unknown. The brain consists of a densely interconnected set of nerve cells, or basic information-processing units, called neurons.

Neural Networks in the Brain

Biological Neural NetworkIn human brain a neuron is a information processing unit

whichreceives several input signals from the environment, computes a new activation level and sends an output signal to the other neurons or body muscles through output links.

• Neural networks exhibit plasticity. • In response to the stimulation pattern,

neurons demonstrate long-term changes in the strength of their connections.

• Neurons also can form new connections with other neurons. Even entire collections of neurons may sometimes migrate from one place to another.

• These mechanisms form the basis for learning in the brain.

MACHINE LEARNING• In general, machine learning

involves adaptive mechanisms that enable computers to learn from experience, learn by example and learn by analogy.

• Learning capabilities can improve the performance of an intelligent system over time.

• Machine learning mechanisms form the basis for adaptive systems.

WHY MACHINE LEARNING?• The techniques that we’ve seen before relies on expert knowledge to set the

rules. Once the rules are set, the decision making is automated.• What happens if the rules become obsolete or new information is gathered?• The change, then needs to happen at a very basic level and needs to be

done manually.• ANNs attempt to automate the process.• The objective is to come up with a model to predict a set of outputs Y <y1,

y2,…, yn> from a given set of inputs X <x1, x2,…, xm> given training dataset with records of the form (X, Y).

• The result must be a function f(X) that approximates Y for values of X not in the dataset.

THE PERCEPTRON

As like as human brain, in ANN the PERCEPTRON is the simplest form of a neural network. It consists of a single “neuron” which computes an output function by assigning weights to each of the links to the n parameters.

How does the Perceptron learn from experience?Weights(w1,w2….) are assigned to inputs of perceptron initially in the range [-0.5,0.5] and then updated to obtain the output consistent with the training examples. Thus weights Are updated and summation of these weights is calculated in linear combiner at each training level.

 

 

Hard Limiters: Step and Sign

UPDATE RULES• Error function for pth training example:

• e(p) = Yd(p) – Y(p); where p = 1, 2, 3, . . .• Update:

• wi(p+1) = wi(p) + α × xi(p) × e(p); • where α is the learning rate, a positive constant less than unity

• Only works on Linearly Separable data.

LIMITATIONS

MULTILAYERED NETWORKS

BACKPROPAGATION• The indices i, j and k here refer to neurons in the input,

hidden and output layers, respectively.

• Input signals, x1, x2, . . . , xn, are propagated through the network from left to right, and error signals, e1, e2, . . .,el, from right to left. The symbol wij denotes the weight for the connection between neuron i in the input layer and neuron j in the hidden layer, and the symbol wjk the weight between neuron j in the hidden layer and neuron k in the output layer.

The error signal at the output of neuron k at iteration p is defined by. Update rule

Outer Layer Hidden Layer(s)

WHEN TO STOP

IMPROVEMENTSA multilayer network, in general, learns much faster when the sigmoidal activation function is represented by a hyperbolic tangent:

We also can accelerate training by including a momentum term in the earlier expression for delta, according to the observations made in Watrous (1987) and Jacobs (1988), the inclusion of momentum in the back-propagation algorithm has a stabilising effect on training:

LINKS

• Intro• Forward Propagation [until 3:45]• Gradient Descent [Cost, Curse of Dimensionality not covered]• Backpropagation

SELF-ORGANISING MAPS(LEARNING WITHOUT A

‘TEACHER’)• Hebb’s Law:

• If two neurons on either side of a connection are activated synchronously, then the weight of that connection is increased.

• If two neurons on either side of a connection are activated asynchronously, then the weight of that connection is decreased.

Forgetting Factor (To allow for the possibility of decreasing connection strength):

HOW TO CHOOSE THE FORGETTING FACTOR

• Forgetting factor specifies the weight decay in a single learning cycle. It usually falls in the interval between 0 and 1.

• If the forgetting factor is 0, the neural network is capable only of strengthening its synaptic weights, and as a result, these weights grow towards infinity. On the other hand, if the forgetting factor is close to 1, the network remembers very little of what it learns.

• Therefore, a rather small forgetting factor should be chosen, typically between 0.01 and 0.1, to allow only a little ‘forgetting’ while limiting the weight growth.