op04 neural networks

Lectured by Ha Hoang Kha, Ph.D.

Ho Chi Minh City University of Technology

Email: [email protected]

Unconstrained Optimization

and Neural Networks

Ho Chi Minh City University of Technology

Faculty of Electrical and Electronics Engineering

Department of Telecommunications

Introduction

Single neuron training

Backpropagation algorithm

Character recognition

Content

Neural Networks 2 H. H. Kha

References

E. K. P. Chong and S. H. Zak, An Introduction to Optimization, Jonh Wiley & Sons, 2001


1. Introduction


Neural networks have found numerous practical

applications: telephone echo cancellation, EEG data

interpretation.

The essence of neural networks lies in the connection

weights between neurons. The selection of these

weights is referred as training or learning.

A popular method for training a neural network is

called the backpropagation algorithm, based on an

unconstrained optimization, and associated gradient

algorithm applied to the problem.

1. Introduction


An artificial neural networks is a circuit composed of

interconnected simple circuit element called neurons.

Each neuron represents a map, typically with multiple

inputs and a single output.

The output of the neuron is a function of the sum of the

inputs.

1. Introduction

The function of the output of the neuron is called the activation function.

The single output of the neuron may be applied as inputs to several other neurons.


1. Introduction

Feedforward neural network: neurons are interconnected in layers, so that the data flow only in one direction.


1. Introduction


2. Single-Neural Training

Consider a single neuron




A gradient method

2. Single-Neural Training-Adaline


Adaptive linear element

3. Backpropagation Algorithm


zj


To solve the above optimization problem, we use a

gradient algorithm with fixed step size


4. Applications: pattern recognition

A pattern is an object, process or event that can be

given a name.

A pattern class (or category) is a set of patterns

sharing common attributes and usually originating

from the same source.

During recognition (or classification) given objects

are assigned to prescribed classes.

A classifier is a machine which performs

classification.


Examples of applications

• Optical Character

Recognition (OCR)

• Biometrics

•Diagnostic systems

•Military applications

• Handwritten: sorting letters by postal code, input device for PDA‘s.

• Printed texts: reading machines for blind people, digitalization of text documents.

• Face recognition, verification, retrieval.

• Finger prints recognition.

• Speech recognition.

• Medical diagnosis: X-Ray, EKG analysis.

• Machine diagnostics, waster detection.

• Automated Target Recognition (ATR).

• Image segmentation and analysis (recognition from aerial or satelite photographs).


Basic concepts

y x

nx

x

x

2

1Feature vector

- A vector of observations (measurements).

- is a point in feature space .

Hidden state

- Cannot be directly measured.

- Patterns with equal hidden state belong to the same class.

Xx

x X

Yy

Task

- To design a classifer (decision rule)

which decides about a hidden state based on an onbservation.

YX :q

Pattern


Components of PR system

Sensors and

preprocessin

g

Feature

extraction Classifier

Class

assignment

• Sensors and preprocessing.

• A feature extraction aims to create discriminative features good for classification.

• A classifier.

• A teacher provides information about hidden state -- supervised learning.

• A learning algorithm sets PR from training examples.

Learning algorithm Teacher

Patter

n


Character recognition

Recognition of both printed and handwritten characters

is a typical domain where neural networks have been

successfully applied.

Optical character recognition systems were among the

first commercial applications of neural networks.

For simplicity, we can limit our task to the recognition

of digits from 0 to 9. Each digit is represented by a 5x9

bit map.

In commercial applications, where a better resolution is

required, at least 16 x16 bit maps are used.


Bit maps for digit recognition


7 8 9 10

12 13 14 15

17 18 19 20

26 27 28 29

31 32 33 34

36 37 38 39

6

2 3 4 51

16

11

22 23 24 2521

42 43 44 4541

35

40

30

Architecture of a neural network

The number of neurons in the input layer is decided by the number of pixels in the bit map. The bit map in our example consists of 45 pixels, and thus we need 45 input neurons.

The output layer has 10 neurons – one neuron for each digit to be recognised.

Complex patterns cannot be detected by a small number of hidden neurons; however too many of them can dramatically increase the computational burden.

Another problem is overfitting. The greater the number of hidden neurons, the greater the ability of the network to recognise existing patterns. However, if the number of hidden neurons is too big, the network might simply memorise all training examples.


Architecture of a neural network


0

1

1

1

0

1

1

1

1

1

41

42

43

44

45

1

2

3

4

5

1

2

3

4

5

0

1

2

3

4

5

6

7

8

9

10

0

0

0

0

0

0

0

1

0

op04 neural networks

Documents