chapter 10 chapter 10 neural network neural network

26
Chapter 10 Neural Network Neural Network

Upload: stuart-murphy

Post on 21-Jan-2016

311 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 10Chapter 10

Neural NetworkNeural NetworkNeural NetworkNeural Network

Page 2: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 92Data Warehouse and Data Mining

Understand how feed-forward networks are used to solve estimation problems. Know how input and output data conversions are performed for neural networks. Understand how feed-forward neural networks learn through backpropagation. Know how genetic learning is applied to train feed-forward neural networks. Know how self-organizing neural networks perform unsupervised clustering. List the strengths and weaknesses of neural networks.

Chapter ObjectiveChapter Objective

Page 3: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 93Data Warehouse and Data Mining

Feed-Forward Neural NetworkFeed-Forward Neural Network

Page 4: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 94Data Warehouse and Data Mining

Feed-Forward Neural NetworkFeed-Forward Neural Network

Page 5: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 95Data Warehouse and Data Mining

Neural Network Training: A Conceptual ViewNeural Network Training: A Conceptual View

Page 6: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 96Data Warehouse and Data Mining

Neural Network Training: A Conceptual ViewNeural Network Training: A Conceptual View

Page 7: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 97Data Warehouse and Data Mining

Neural Network Training: A Conceptual ViewNeural Network Training: A Conceptual View

Page 8: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 98Data Warehouse and Data Mining

Neural Network ExplanationNeural Network Explanation

Sensitivity analysis is a technique that has been successfully applied to gain insight into the effect individual attributes have on neural network output.

The general process consists of the following steps:

1. Divide the data into a training set and a test dataset.2. Train the network with the training data.

Page 9: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 99Data Warehouse and Data Mining

Neural Network ExplanationNeural Network Explanation

3. Use the test set data to create a new instance I. Each attribute value for I is the average of all attribute values within the test data.

4. For each attribute:

a. Vary the attribute value within instance I and present the modification of I to the network for classification.

b. Determine the effect the variations have on the output of the neural network.

c. The relative importance of each attribute is measured by the effect of attribute variations on network output.

Page 10: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 910Data Warehouse and Data Mining

General ConsiderationsGeneral Considerations

The following is a partial list of choices that affect the performance of a neural network model:

•What input attributes will be used to build the network?

•How will the network output be represented?

•How many hidden layers should the network contain?

•How many nodes should there be in each hidden layer?

•What condition will terminate network training?

Page 11: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 911Data Warehouse and Data Mining

Neural Network Training: A Detailed ViewNeural Network Training: A Detailed View

Page 12: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 912Data Warehouse and Data Mining

Neural Networks

• Advantages– prediction accuracy is generally high– robust, works when training examples contain

errors– output may be discrete, real-valued, or a vector of

several discrete or real-valued attributes– fast evaluation of the learned target function

• Criticism– long training time– difficult to understand the learned function

(weights)– not easy to incorporate domain knowledge

Page 13: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 913Data Warehouse and Data Mining

A Neuron

• The n-dimensional input vector x is mapped into variable y by means of the scalar product and a nonlinear function mapping

k-

f

weighted sum

Inputvector x

output y

Activationfunction

weightvector w

w0

w1

wn

x0

x1

xn

Page 14: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 9Data Warehouse and Data Mining

Network Training• The ultimate objective of training

– obtain a set of weights that makes almost all the tuples in the training data classified correctly

• Steps– Initialize weights with random values – Feed the input tuples into the network..... one by one– For each unit

• Compute the net input to the unit as a linear combination of all the inputs to the unit

• Compute the output value using the activation function

• Compute the error• Update the weights and the bias

Page 15: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 9Data Warehouse and Data Mining

Multi-Layer Perceptron

Output nodes

Input nodes

Hidden nodes

Output vector

Input vector: xi

wij i

jiijj OwI

jIje

O

1

1

Page 16: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 918Data Warehouse and Data Mining

•A neural network is parallel computing system of several interconnected processor nodes.

•The input to individual network nodes is restricted to numeric values falling in the closed interval range [0,1].

•Because of this, categorical data must be transformed prior to network training.

Chapter SummaryChapter Summary

Page 17: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 919Data Warehouse and Data Mining

Chapter SummaryChapter Summary•Developing a neural network involves first training the

network to carry out the desired computations and then applying the trained network to solve new problems.

• During the learning phase, training data is used to modify the connection weights between pairs of nodes so as to obtain a best result for the output node (s).

•The feed-forward neural network architecture is commonly used for supervised learning.

•Feed-forward neural networks contain a set of layered nodes and weighted connections between nodes in adjacent layers.

Page 18: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 920Data Warehouse and Data Mining

• Feed-forward neural networks are often trained using a backpropagation learning scheme.

• Backpropagation learning works by making modifications in weight values starting at the output layer then moving backward through the hidden layers of the network.

• Genetic learning can also be applied to train feed-forward networks.

Chapter SummaryChapter Summary

Page 19: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 921Data Warehouse and Data Mining

Chapter SummaryChapter Summary

•The self-organizing Kohonen neural network architecture is a popular model for unsupervised clustering.

•A self-organizing neural network learns by having several output nodes complete for the training instances.

• For each instance, the output node whose weight vectors most closely match the attribute values of the input instance is the winning node.

Page 20: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 922Data Warehouse and Data Mining

Chapter SummaryChapter Summary

•As a result, the winning node has its associated input weights modified to more closely match the current training instance.

•When unsupervised learning is complete, output nodes winning the most instances are saved.

•After this, test data is applied and the clusters formed by the test set data are analyzed to help determine the meaning of what has been found.

Page 21: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 923Data Warehouse and Data Mining

Chapter SummaryChapter Summary•A central issue surrounding neural network is their

inability to explain what has been learned.

•Despite this, neural network have been successfully applied to solve problems in both the business and scientific world.

•Although we have discussed the most popular neural network models, several other architectures and learning rules have been developed.

•Jain, Mao, and Mohiuddin (1996) provide a good starting point for learning more about neural networks.

Page 22: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 924Data Warehouse and Data Mining

Key TermsKey Terms

Average member technique. An unsupervised clustering neural network explanation technique where the most typical member of each cluster is computed by finding the average value for each class attribute.

Backpropagation learning. A training method used with many feed-forward networks that works by making modifications in weight values starting at the output layer then moving backward through the hidden layer.

Delta rule. A neural network learning rule designed to minimize the sum of squared errors between computed and target network output.

Page 23: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 925Data Warehouse and Data Mining

Key TermsKey Terms

Epoch. One complete pass of the training data through a neural network.

Feed-forward neural network. A neural network architecture where all weights at one layer are directed toward nodes at the next network layer. Weights do not cycle back as inputs to previous layers.

Fully connected. A neural network structure where all nodes at one layer of the network are connected to all nodes in the next layer.

Kohonen network. A two-layer neural network used for unsupervised clustering.

Page 24: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 926Data Warehouse and Data Mining

Key TermsKey Terms

Neural network. A parallel computing system consisting of several interconnected processors.

Neurode. A neural network processor node. Several neurodes are connected to form a complete neural network structure.

Sensitivity analysis. A neural network explanation technique that allows us to determine a rank ordering for the relative importance of individual attributes.

Sigmoid function. One of several commonly used neural network evaluation functions. The sigmoid function is continuous and outputs a value between 0 or 1.

Page 25: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 927Data Warehouse and Data Mining

Key TermsKey Terms

Linearly separable.Linearly separable. Two classes, A and B, are said Two classes, A and B, are said to be linearly separable if a straight line can be to be linearly separable if a straight line can be drawn to separate the instances of class B.drawn to separate the instances of class B.

Perceptron neural network.Perceptron neural network. A simple feed-forward A simple feed-forward neural network architecture consisting of an neural network architecture consisting of an input layer and a single output layer.input layer and a single output layer.

Page 26: Chapter 10 Chapter 10 Neural Network Neural Network

Chapter 928Data Warehouse and Data Mining

Reference

Data Mining: Concepts and Techniques (Chapter 7 Slide for textbook), Jiawei Han and Micheline Kamber, Intelligent Database Systems Research Lab, School of Computing Science, Simon Fraser University, Canada