neural network architectures

Download Neural Network Architectures

Post on 05-Jan-2016

53 views

Category:

Documents

1 download

Embed Size (px)

DESCRIPTION

Neural Network Architectures. Aydn Ula. 02 December 2004. ulasmehm @boun.edu.tr. Outline Of Presentation. Introduction Neural Networks Neural Network Architectures Conclusions. Introduction. Some numbers The human brain contains about 10 billion nerve cells (neurons) - PowerPoint PPT Presentation

TRANSCRIPT

  • Neural Network ArchitecturesAydn Ula02 December 2004ulasmehm@boun.edu.tr

    Aydn Ula

  • Outline Of PresentationIntroductionNeural NetworksNeural Network ArchitecturesConclusions

    Aydn Ula

  • IntroductionSome numbersThe human brain contains about 10 billion nerve cells (neurons)Each neuron is connected to the others through 10000 synapses

    Brain as a computational unit It can learn, reorganize from experienceIt adapts to the environment It is robust and fault tolerantFast computations with too much individual computational units

    Aydn Ula

  • IntroductionTaking the nature as a model. Consider the neuron as a PEA neuron hasInput (dendrites)Output (the axon)The information circulates from the dendrites to the axon via the cell bodyAxon connects to dendrites via synapsesStrength of synapses changeSynapses may be excitatory or inhibitory

    Aydn Ula

  • Perceptron (Artificial Neuron)Definition : Non linear, parameterized function with restricted output range

    Aydn Ula

  • Activation FunctionsLinearSigmoidHyperbolic tangent

    Aydn Ula

  • Neural NetworksA mathematical model to solve engineering problemsGroup of highly connected neurons to realize compositions of non linear functionsTasksClassificationClusteringRegression According to input flowFeed forward Neural NetworksRecurrent Neural Networks

    Aydn Ula

  • Feed Forward Neural NetworksThe information is propagated from the inputs to the outputsTime has no role (Acyclic, no feedbacks from outputs to inputs)

    Aydn Ula

  • Recurrent NetworksArbitrary topologiesCan model systems with internal states (dynamic ones)Delays can be modeledMore difficult to trainProblematic performanceStable Outputs may be more difficult to evaluateUnexpected behavior (oscillation, chaos, )

    x1x2

    Aydn Ula

  • LearningThe procedure that consists in estimating the parameters of neurons (setting up the weights) so that the whole network can perform a specific task.2 types of learningSupervised learningUnsupervised learningThe Learning process (supervised)Present the network a number of inputs and their corresponding outputs (Training)See how closely the actual outputs match the desired onesModify the parameters to better approximate the desired outputsSeveral passes over the data

    Aydn Ula

  • Supervised LearningThe real outputs of the model for the given inputs is known in advance. The networks task is to approximate those outputs.A Supervisor provides examples and teach the neural network how to fulfill a certain task

    Aydn Ula

  • Unsupervised learningGroup typical input data according to some function.Data clusteringNo need of a supervisor Network itself finds the correlations between the dataExamples:Kohonen feature maps (SOM)

    Aydn Ula

  • Properties of Neural NetworksSupervised networks are universal approximators (Non recurrent networks)Can act asLinear Approximator (Linear Perceptron)Nonlinear Approximator (Multi Layer Perceptron)

    Aydn Ula

  • Other PropertiesAdaptivityAdapt weights to the environment easilyAbility to generalizeMay provide against lack of data Fault toleranceNot too much degradation of performances if damaged The information is distributed within the entire net.

    Aydn Ula

  • An Example Regression

    Aydn Ula

  • Example ClassificationHandwritten digit recognition16x16 bitmap representationConverted to 1x256 bit vector7500 points on training set3500 points on test set0000000001100000000000011010000000000001000000000000001000000000000001000000000000001000000000000000100000000000000010000000000000001000000000000001000111110000000101100001100000011000000010000001100000001000000100000000100000001000000100000000011111110000

    Aydn Ula

  • TrainingTry to minimize an error or cost functionBackpropogation algorithmGradient DescentLearn the weights of the networkUpdate the weights according to the error function

    Aydn Ula

  • ApplicationsHandwritten Digit RecognitionFace recognitionTime series predictionProcess identificationProcess controlOptical character recognitionEtc

    Aydn Ula

  • Neural NetworksNeural networks are statistical toolsAdjust non linear functions to accomplish a taskNeed of multiple and representative examples but fewer than in other methodsNeural networks can model static (FF) and dynamic (RNN) tasksNNs are good classifiers BUTGood representations of data have to be formulatedTraining vectors must be statistically representative of the entire input spaceThe use of NN needs a good comprehension of the problem

    Aydn Ula

  • Implementation of Neural NetworksGeneric architectures (PCs etc)Specific Neuro-HardwareDedicated circuits

    Aydn Ula

  • Generic architecturesConventional microprocessorsIntel Pentium, Power PC, etc AdvantagesHigh performances (clock frequency, etc)CheapSoftware environment available (NN tools, etc)DrawbacksToo generic, not optimized for very fast neural computations

    Aydn Ula

  • Classification of HardwareNN HardwareNeurochipsSpecial PurposeGeneral Purpose (Ni1000, L - Neuro)NeuroComputersSpecial Purpose (CNAPS, Synapse)General Purpose

    Aydn Ula

  • Specific Neuro-hardware circuitsCommercial chips CNAPS, Synapse, etc.AdvantagesCloser to the neural applicationsHigh performances in terms of speedDrawbacksNot optimized to specific applicationsAvailabilityDevelopment tools

    Aydn Ula

  • CNAPSSIMDOne instruction sequencing and control unitProcessor nodes (PN)Single dimensional array (only right or left nodes)

    Aydn Ula

  • CNAPS 1064

    Aydn Ula

  • CNAPS

    Aydn Ula

  • Dedicated circuitsA system where the functionality is buried in the hardware. For specific applications only not changeableAdvantagesOptimized for a specific applicationHigher performances than the other systemsDrawbacksHigh development costs in terms of time and money

    Aydn Ula

  • What type of hardware to be used in dedicated circuits ?Custom circuitsASIC (Application-Specific Integrated Circuit)Necessity to have good knowledge of the hardware designFixed architecture, hardly changeableOften expensive Programmable logicValuable to implement real time systemsFlexibilityLow development costsLower performances compared to ASIC (Frequency, etc.)

    Aydn Ula

  • Programmable logicField Programmable Gate Arrays (FPGAs)Matrix of logic cells Programmable interconnectionAdditional features (internal memories + embedded resources like multipliers, etc.)ReconfigurabilityWe can change the configurations as many times as desired

    Aydn Ula

  • Real Time SystemsExecution of applications with time constraints.Hard real-time systemsDigital fly-by-wire control system of an aircraft: No lateness is accepted. The lives of people depend on the correct working of the control system of the aircraft.Soft real-time systemsVending machine: Accept lower performance for lateness, it is not catastrophic when deadlines are not met. It will take longer to handle one client with the vending machine.

    Aydn Ula

  • Real Time Systemsms scale real time systemConnectionist retina for image processingArtificial Retina: combining an image sensor with a parallel architectures scale real time systemLevel 1 trigger in a HEP experiment

    Aydn Ula

  • Connectionist RetinaIntegration of a neural network in an artificial retinaScreenMatrix of Active Pixel sensorsCAN 8 bits ADC converter 256 levels of greyProcessing ArchitectureParallel system where neural networks are implemented

    ProcessingArchitecture

    Aydn Ula

  • Maharadja Processing ArchitectureMicro-controllerGeneric architecture executing sequential cost with low power consumptionMemory 256 Kbytes shared between processor, PEs, inputStore the network parametersUNE (Unit Neural SIMD Completely pipelined 16 bit internal data bus)Processors to compute the neurons outputsCommand bus manages all different operators in UNEInput/Output moduleData acquisition and storage of intermediate results

    Micro-controllerSequencer Command busInput/OutputunitInstruction BusUNE-0UNE-1UNE-2UNE-3MMMM

    Aydn Ula

  • Level 1 trigger in a HEP experimentHigh Energy Physics (Particle Physics)Neural networks have provided interesting results as triggers in HEP.Level 2 : H1 experiment 10 20 s Level 1 : Dirac experiment 2 s Particle RecognitionHigh timing constraints (in terms of latency and data throughput)

    Aydn Ula

  • Neural Network architecture....64128Execution time : ~500 ns

    Weights coded in 16 bitsStates coded in 8 bits with data arriving every BC=25ns4Electrons, tau, hadrons, jets

    Aydn Ula

  • Very Fast ArchitectureTanHPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPETanHTanHTanHACCACCACCACC256 PEsMatrix of n*m matrix elementsControl unitI/O moduleTanH are stored in LUTs1 matrix row computes a neuronThe results is back-propagated to calculate the output layer

    I/O moduleControl unit

    Aydn Ula

  • PE architectureXAccumulatorMultiplierWeights memInput data816Addr gen +Data incmd busControl ModuleData out

    Aydn Ula

  • Neuro-hardware todayGeneric Real time applicationsMicroprocessors technology (PCs, computers, i.e. software) is sufficient to implement most of neural applications in real-time (ms or sometimes s scale) This solution is cheapVery easy to manageConstrained Real time applicationsIt still remains specific applications where powerful computations are needed e.g. particle physicsIt still remains applications where other constraints have to be taken into consideration (Consumption, proximity of sensors, mixed integration, etc.)

    Aydn Ula

  • ClusteringIdea

Recommended

View more >