# neural network architectures

Post on 05-Jan-2016

53 views

Embed Size (px)

DESCRIPTION

Neural Network Architectures. Aydn Ula. 02 December 2004. ulasmehm @boun.edu.tr. Outline Of Presentation. Introduction Neural Networks Neural Network Architectures Conclusions. Introduction. Some numbers The human brain contains about 10 billion nerve cells (neurons) - PowerPoint PPT PresentationTRANSCRIPT

Neural Network ArchitecturesAydn Ula02 December 2004ulasmehm@boun.edu.tr

Aydn Ula

Outline Of PresentationIntroductionNeural NetworksNeural Network ArchitecturesConclusions

Aydn Ula

IntroductionSome numbersThe human brain contains about 10 billion nerve cells (neurons)Each neuron is connected to the others through 10000 synapses

Brain as a computational unit It can learn, reorganize from experienceIt adapts to the environment It is robust and fault tolerantFast computations with too much individual computational units

Aydn Ula

IntroductionTaking the nature as a model. Consider the neuron as a PEA neuron hasInput (dendrites)Output (the axon)The information circulates from the dendrites to the axon via the cell bodyAxon connects to dendrites via synapsesStrength of synapses changeSynapses may be excitatory or inhibitory

Aydn Ula

Perceptron (Artificial Neuron)Definition : Non linear, parameterized function with restricted output range

Aydn Ula

Activation FunctionsLinearSigmoidHyperbolic tangent

Aydn Ula

Neural NetworksA mathematical model to solve engineering problemsGroup of highly connected neurons to realize compositions of non linear functionsTasksClassificationClusteringRegression According to input flowFeed forward Neural NetworksRecurrent Neural Networks

Aydn Ula

Feed Forward Neural NetworksThe information is propagated from the inputs to the outputsTime has no role (Acyclic, no feedbacks from outputs to inputs)

Aydn Ula

Recurrent NetworksArbitrary topologiesCan model systems with internal states (dynamic ones)Delays can be modeledMore difficult to trainProblematic performanceStable Outputs may be more difficult to evaluateUnexpected behavior (oscillation, chaos, )

x1x2

Aydn Ula

LearningThe procedure that consists in estimating the parameters of neurons (setting up the weights) so that the whole network can perform a specific task.2 types of learningSupervised learningUnsupervised learningThe Learning process (supervised)Present the network a number of inputs and their corresponding outputs (Training)See how closely the actual outputs match the desired onesModify the parameters to better approximate the desired outputsSeveral passes over the data

Aydn Ula

Supervised LearningThe real outputs of the model for the given inputs is known in advance. The networks task is to approximate those outputs.A Supervisor provides examples and teach the neural network how to fulfill a certain task

Aydn Ula

Unsupervised learningGroup typical input data according to some function.Data clusteringNo need of a supervisor Network itself finds the correlations between the dataExamples:Kohonen feature maps (SOM)

Aydn Ula

Properties of Neural NetworksSupervised networks are universal approximators (Non recurrent networks)Can act asLinear Approximator (Linear Perceptron)Nonlinear Approximator (Multi Layer Perceptron)

Aydn Ula

Other PropertiesAdaptivityAdapt weights to the environment easilyAbility to generalizeMay provide against lack of data Fault toleranceNot too much degradation of performances if damaged The information is distributed within the entire net.

Aydn Ula

An Example Regression

Aydn Ula

Example ClassificationHandwritten digit recognition16x16 bitmap representationConverted to 1x256 bit vector7500 points on training set3500 points on test set0000000001100000000000011010000000000001000000000000001000000000000001000000000000001000000000000000100000000000000010000000000000001000000000000001000111110000000101100001100000011000000010000001100000001000000100000000100000001000000100000000011111110000

Aydn Ula

TrainingTry to minimize an error or cost functionBackpropogation algorithmGradient DescentLearn the weights of the networkUpdate the weights according to the error function

Aydn Ula

ApplicationsHandwritten Digit RecognitionFace recognitionTime series predictionProcess identificationProcess controlOptical character recognitionEtc

Aydn Ula

Neural NetworksNeural networks are statistical toolsAdjust non linear functions to accomplish a taskNeed of multiple and representative examples but fewer than in other methodsNeural networks can model static (FF) and dynamic (RNN) tasksNNs are good classifiers BUTGood representations of data have to be formulatedTraining vectors must be statistically representative of the entire input spaceThe use of NN needs a good comprehension of the problem

Aydn Ula

Implementation of Neural NetworksGeneric architectures (PCs etc)Specific Neuro-HardwareDedicated circuits

Aydn Ula

Generic architecturesConventional microprocessorsIntel Pentium, Power PC, etc AdvantagesHigh performances (clock frequency, etc)CheapSoftware environment available (NN tools, etc)DrawbacksToo generic, not optimized for very fast neural computations

Aydn Ula

Classification of HardwareNN HardwareNeurochipsSpecial PurposeGeneral Purpose (Ni1000, L - Neuro)NeuroComputersSpecial Purpose (CNAPS, Synapse)General Purpose

Aydn Ula

Specific Neuro-hardware circuitsCommercial chips CNAPS, Synapse, etc.AdvantagesCloser to the neural applicationsHigh performances in terms of speedDrawbacksNot optimized to specific applicationsAvailabilityDevelopment tools

Aydn Ula

CNAPSSIMDOne instruction sequencing and control unitProcessor nodes (PN)Single dimensional array (only right or left nodes)

Aydn Ula

CNAPS 1064

Aydn Ula

CNAPS

Aydn Ula

Dedicated circuitsA system where the functionality is buried in the hardware. For specific applications only not changeableAdvantagesOptimized for a specific applicationHigher performances than the other systemsDrawbacksHigh development costs in terms of time and money

Aydn Ula

What type of hardware to be used in dedicated circuits ?Custom circuitsASIC (Application-Specific Integrated Circuit)Necessity to have good knowledge of the hardware designFixed architecture, hardly changeableOften expensive Programmable logicValuable to implement real time systemsFlexibilityLow development costsLower performances compared to ASIC (Frequency, etc.)

Aydn Ula

Programmable logicField Programmable Gate Arrays (FPGAs)Matrix of logic cells Programmable interconnectionAdditional features (internal memories + embedded resources like multipliers, etc.)ReconfigurabilityWe can change the configurations as many times as desired

Aydn Ula

Real Time SystemsExecution of applications with time constraints.Hard real-time systemsDigital fly-by-wire control system of an aircraft: No lateness is accepted. The lives of people depend on the correct working of the control system of the aircraft.Soft real-time systemsVending machine: Accept lower performance for lateness, it is not catastrophic when deadlines are not met. It will take longer to handle one client with the vending machine.

Aydn Ula

Real Time Systemsms scale real time systemConnectionist retina for image processingArtificial Retina: combining an image sensor with a parallel architectures scale real time systemLevel 1 trigger in a HEP experiment

Aydn Ula

Connectionist RetinaIntegration of a neural network in an artificial retinaScreenMatrix of Active Pixel sensorsCAN 8 bits ADC converter 256 levels of greyProcessing ArchitectureParallel system where neural networks are implemented

ProcessingArchitecture

Aydn Ula

Maharadja Processing ArchitectureMicro-controllerGeneric architecture executing sequential cost with low power consumptionMemory 256 Kbytes shared between processor, PEs, inputStore the network parametersUNE (Unit Neural SIMD Completely pipelined 16 bit internal data bus)Processors to compute the neurons outputsCommand bus manages all different operators in UNEInput/Output moduleData acquisition and storage of intermediate results

Micro-controllerSequencer Command busInput/OutputunitInstruction BusUNE-0UNE-1UNE-2UNE-3MMMM

Aydn Ula

Level 1 trigger in a HEP experimentHigh Energy Physics (Particle Physics)Neural networks have provided interesting results as triggers in HEP.Level 2 : H1 experiment 10 20 s Level 1 : Dirac experiment 2 s Particle RecognitionHigh timing constraints (in terms of latency and data throughput)

Aydn Ula

Neural Network architecture....64128Execution time : ~500 ns

Weights coded in 16 bitsStates coded in 8 bits with data arriving every BC=25ns4Electrons, tau, hadrons, jets

Aydn Ula

Very Fast ArchitectureTanHPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPETanHTanHTanHACCACCACCACC256 PEsMatrix of n*m matrix elementsControl unitI/O moduleTanH are stored in LUTs1 matrix row computes a neuronThe results is back-propagated to calculate the output layer

I/O moduleControl unit

Aydn Ula

PE architectureXAccumulatorMultiplierWeights memInput data816Addr gen +Data incmd busControl ModuleData out

Aydn Ula

Neuro-hardware todayGeneric Real time applicationsMicroprocessors technology (PCs, computers, i.e. software) is sufficient to implement most of neural applications in real-time (ms or sometimes s scale) This solution is cheapVery easy to manageConstrained Real time applicationsIt still remains specific applications where powerful computations are needed e.g. particle physicsIt still remains applications where other constraints have to be taken into consideration (Consumption, proximity of sensors, mixed integration, etc.)

Aydn Ula

ClusteringIdea

Recommended