Neural Net UpdateNeural Net UpdateDave BaileyDave Bailey

What’s NewWhat’s New

You can now save and load datasets from a filee.g. saving the dataset:

You can now save and load datasets from a filee.g. saving the dataset:

NeuralNetDataSet myDataSet;

… // Fill the DataSet

std::ofstream ofs;ofs.open(“myfile.dat”,std::ios_base::out|std::ios_base::trunc);if (ofs.is_open()) { ofs << myDataSet; // Writes the dataset to the stream

ofs.close();}

What’s New (2)What’s New (2)

Loading a previously saved dataset:Loading a previously saved dataset:

std::string filename(“myfile.dat”);NeuralNetDataSet myDataSet(filename);

DataSets are saved to a plain text fileSo be warned, the files can be quite

large if you have a lot of data in themYou can use as many different

datasets as you like to train the network

DataSets are saved to a plain text fileSo be warned, the files can be quite

large if you have a lot of data in themYou can use as many different

datasets as you like to train the network

Feature FixesFeature Fixes

There was a problem with part of the conjugate gradients training algorithmShowed up as big jumps in the error

function whilst trainingDue to an oversight (and a small bug)

in the linear search that optimises the training step length

There was a problem with part of the conjugate gradients training algorithmShowed up as big jumps in the error

function whilst trainingDue to an oversight (and a small bug)

in the linear search that optimises the training step length

Search TechniqueSearch Technique

Remember that the network “learns” by

The conjugate gradient method attempts to optimise λ and μ for each epoch

Remember that the network “learns” by

The conjugate gradient method attempts to optimise λ and μ for each epoch

1

1 1 1 1

where

and

(just the derivative of the error function)

L L L L L Lij i i i j ij

L L L L Li j j j ji

j

Li L

i

w f u o w prev

f u w

E

o

Linear SearchLinear Search

Optimisation is performed using a linear search along a direction in “weight space”, looking for the minimum of the error function

Optimisation is performed using a linear search along a direction in “weight space”, looking for the minimum of the error function

Slope -σψ’(0)Slope σψ’(0)

ψ(0)+step*μψ’(0)

Slope ψ’(0)

μ,σ are search parameters

Try to end up here

Problem and SolutionProblem and Solution

Basic assumption was that the minimum would be found for positive step lengths Not true in real life as the starting point

could be anywhere in weight space Line search is now bi-directional

i.e. can search backwards up the direction vector

Convergence much more stable and the problem of jumping error function is gone

Basic assumption was that the minimum would be found for positive step lengths Not true in real life as the starting point

could be anywhere in weight space Line search is now bi-directional

i.e. can search backwards up the direction vector

Convergence much more stable and the problem of jumping error function is gone