ann intro jan2010

8/11/2019 ANN Intro Jan2010

1/29

1

Artificial Neural Networks:An Introduction

S. Bapi Raju

Dept. of Computer and

Information Sciences,University of Hyderabad


2/29

ANN-Intro (Jan 2010)

2 of 29

OUTLINEBiological Neural Networks

Applications of Artificial Neural Networks

Taxonomy of Artificial Neural Networks

Supervised and Unsupervised Artificial NeuralNetworks

Basis function and Activation function

Learning Rules

Applications OCR, Load Forecasting, Condition Monitoring


3/29


3 of 29

Biological Neural NetworksStudy of Neural Networks originates in biological systems

Human Brain:contains over 100 billion neurons, number ofsynapses is approximately 1000 times that

in electronic circuit terms: synaptic fan-in fan-out is 1000,

switching time of a neuron is order of milliseconds

But on a face recognition problem brain beats fastest

supercomputer in terms of number of cycles of computationto arrive at answer

Neuronal Structure

Cellbody

Dendrites for inputAxon carries output to otherdendritesSynapse-where they meetActivation signal (voltage)travels along axon


4/29


4 of 29

Need for ANNStandard Von Neumman Computing as existing

presently has some shortcomings.

Following are some desirable characteristics in ANN

Learning AbilityGeneralization and Adaptation

Distributed and Parallel representation

Fault Tolerance

Low Power requirementsPerformance comes not just from the computational

elements themselves but the manner of networkedinterconnectedness of the decision process.


5/29


5 of 29

VonNeumann

versus

BiologicalComputer


6/29


6 of 29

ANN Applications

Pattern Classification Speech Recognition, ECG/EEG classification, OCR


7/29


7 of 29

ANN Applications

Clustering/Categorization Data mining, data compression


8/29


8 of 29

ANN Applications

Function Approximation Noisy arbitrary function needs to be approximated


9/29


9 of 29

ANN Applications

Prediction/Forecasting Given a function of time, predict the function values

for future time values, used in weather predictionand stock market predictions


10/29


10 of 29

ANN Applications

Optimization Several scientific and other problems can be reduced

to an optimization problem like the TravelingSalesman Problem (TSP)


11/29


11 of 29

ANN Applications

Content Based Retrieval Given the partial description of an object retrieve the

objects that match this


12/29

f


13/29


13 of 29

Characteristics of ANNBiologically inspired computational units

Also called as Connectionist Models orConnectionist Architectures

Large number of simple processing elementsVery large number of weighted connectionsbetween elements. Information in thenetwork is encoded in the weights learned by

the connectionsParallel and distributed control

Connection weights are learned by automatictraining techniques

14 f 29


14/29


14 of 29

Artifical Neuron Working Model

Objective is to create a model of functioning ofbiological neuron to aid computation

All signals at synapses are

summed i.e. all the excitatory

and inhibitory influences and

represented by a net value h(.)

If the excitatory influences are

dominant, then the neuron fires,

this is modeled by a simple

threshold function (.)

Certain inputs are fixed biases

Output yleads to other

neurons

McCulloch Pitts Model

15 f 29


15/29


15 of 29

More about the Model

Activation Functions play a key role Simple thresholding (hard limiting)

Squashing Function (sigmoid)

Gaussian Function Linear Function

Biases are also learnt

16 f 29


16/29


16 of 29

Different Kinds of NetworkArchitectures

17 f 29


17/29


17 of 29

Learning AbilityMere Architecture is insufficient

Learning Techniques also need to be formulated

Learning is a process where connection weights are

adjustedLearning is done by training from labeled examples.This is the most powerful and useful aspect of neuralnetworks in their use as Black Box classifiers.

Most commonly an input-output relationshipis learntLearning Paradigm needs to be specified

Weight update in learning rules must be specified

Learning Algorithm specifies step by step procedure

18 of 29


18/29


18 of 29

Learning TheoryMajor Factors Learning Capacity: This concerns the number of

patterns that can be learntand the functionsand kinds of decision boundaries that can be

formed Sample Complexity: This concerns the number of

the samples needed to learn withgeneralization. Overfitting problem is to beavoided

Computational Complexity: This concerns thecomputation time neededto learn theconcepts embedded in the training samples.Generally the computational complexity of learning

is high.


19/29

20 of 29


20/29


20 of 29

Major Learning Rules

Error Correction:Error signal (dy)used to adjust the

weights so thateventually desiredoutput disproduced

Perceptron SolvingAND Problem

21 of 29


21/29


21 of 29


Error Correction: in Mutlilayer Feedforward Network

Geometric interpretation of the role of hidden units in a 2D input space

22 of 29


22/29


22 of 29

Major Learning RulesHebbian:weights are adjusted by afactor proportional to the activities ofthe neurons associated

OrientationSelectivity of aSingle HebbianNeuron

23 of 29


23/29


23 of 29


Competitive Learning: winner take all

(a) Before Learning (b) After Learning

24 of 29


24/29


24 of 29

Summary of ANN Algorithms

25 of 29


25/29


25 of 29

26 of 29


26/29


26 of 29

Application to OCR SystemThe main problemin the HandwrittenLetter recognition isthat characters withvariation inthickness shape,rotation anddifferent nature ofstrokes need to berecognized as ofbeing in thedifferent categoriesfor each letter.

Sufficient number ofsample training datais required for eachcharacter to train

the networks

A Sample set of characters in the NIST Data

27 of 29


27/29


27 of 29

OCRProcess

28 of 29


28/29


28 of 29

OCR Example (continued)

Two schemes shown at right

First makes use of the

feature extractors

Second uses the image

pixels directly

29 of 29


29/29


References

A. K. Jain, J.Mao, K.Mohiuddin, ANN aTutorial, IEEE Computer, 1996 March, pp 31-44 (Figures and Tables taken from this reference)

B. Yegnanarayana,Artificial Neural Networks,Prentice Hall of India, 2001.

Y. M. Zurada, Inroduction toArtificial Neural

Systems, Jaico, 1999.MATLAB neural networks toolbox and manual

ann intro jan2010

Documents