Upload File

neural network architectures

43
Neural Network Neural Network Architectures Architectures Aydın Ulaş 02 December 2004 [email protected]

Upload: martha

Post on 05-Jan-2016

79 views

Category:

Documents


2 download

Report

Tags:

DESCRIPTION

Neural Network Architectures. Aydın Ulaş. 02 December 2004. ulasmehm @boun.edu.tr. Outline Of Presentation. Introduction Neural Networks Neural Network Architectures Conclusions. Introduction. Some numbers… The human brain contains about 10 billion nerve cells (neurons) - PowerPoint PPT Presentation

TRANSCRIPT

  • Neural Network ArchitecturesAydn Ula02 December [email protected]

    Aydn Ula

  • Outline Of PresentationIntroductionNeural NetworksNeural Network ArchitecturesConclusions

    Aydn Ula

  • IntroductionSome numbersThe human brain contains about 10 billion nerve cells (neurons)Each neuron is connected to the others through 10000 synapses

    Brain as a computational unit It can learn, reorganize from experienceIt adapts to the environment It is robust and fault tolerantFast computations with too much individual computational units

    Aydn Ula

  • IntroductionTaking the nature as a model. Consider the neuron as a PEA neuron hasInput (dendrites)Output (the axon)The information circulates from the dendrites to the axon via the cell bodyAxon connects to dendrites via synapsesStrength of synapses changeSynapses may be excitatory or inhibitory

    Aydn Ula

  • Perceptron (Artificial Neuron)Definition : Non linear, parameterized function with restricted output range

    Aydn Ula

  • Activation FunctionsLinearSigmoidHyperbolic tangent

    Aydn Ula

  • Neural NetworksA mathematical model to solve engineering problemsGroup of highly connected neurons to realize compositions of non linear functionsTasksClassificationClusteringRegression According to input flowFeed forward Neural NetworksRecurrent Neural Networks

    Aydn Ula

  • Feed Forward Neural NetworksThe information is propagated from the inputs to the outputsTime has no role (Acyclic, no feedbacks from outputs to inputs)

    Aydn Ula

  • Recurrent NetworksArbitrary topologiesCan model systems with internal states (dynamic ones)Delays can be modeledMore difficult to trainProblematic performanceStable Outputs may be more difficult to evaluateUnexpected behavior (oscillation, chaos, )

    x1x2

    Aydn Ula

  • LearningThe procedure that consists in estimating the parameters of neurons (setting up the weights) so that the whole network can perform a specific task.2 types of learningSupervised learningUnsupervised learningThe Learning process (supervised)Present the network a number of inputs and their corresponding outputs (Training)See how closely the actual outputs match the desired onesModify the parameters to better approximate the desired outputsSeveral passes over the data

    Aydn Ula

  • Supervised LearningThe real outputs of the model for the given inputs is known in advance. The networks task is to approximate those outputs.A Supervisor provides examples and teach the neural network how to fulfill a certain task

    Aydn Ula

  • Unsupervised learningGroup typical input data according to some function.Data clusteringNo need of a supervisor Network itself finds the correlations between the dataExamples:Kohonen feature maps (SOM)

    Aydn Ula

  • Properties of Neural NetworksSupervised networks are universal approximators (Non recurrent networks)Can act asLinear Approximator (Linear Perceptron)Nonlinear Approximator (Multi Layer Perceptron)

    Aydn Ula

  • Other PropertiesAdaptivityAdapt weights to the environment easilyAbility to generalizeMay provide against lack of data Fault toleranceNot too much degradation of performances if damaged The information is distributed within the entire net.

    Aydn Ula

  • An Example Regression

    Aydn Ula

  • Example ClassificationHandwritten digit recognition16x16 bitmap representationConverted to 1x256 bit vector7500 points on training set3500 points on test set0000000001100000000000011010000000000001000000000000001000000000000001000000000000001000000000000000100000000000000010000000000000001000000000000001000111110000000101100001100000011000000010000001100000001000000100000000100000001000000100000000011111110000

    Aydn Ula

  • TrainingTry to minimize an error or cost functionBackpropogation algorithmGradient DescentLearn the weights of the networkUpdate the weights according to the error function

    Aydn Ula

  • ApplicationsHandwritten Digit RecognitionFace recognitionTime series predictionProcess identificationProcess controlOptical character recognitionEtc

    Aydn Ula

  • Neural NetworksNeural networks are statistical toolsAdjust non linear functions to accomplish a taskNeed of multiple and representative examples but fewer than in other methodsNeural networks can model static (FF) and dynamic (RNN) tasksNNs are good classifiers BUTGood representations of data have to be formulatedTraining vectors must be statistically representative of the entire input spaceThe use of NN needs a good comprehension of the problem

    Aydn Ula

  • Implementation of Neural NetworksGeneric architectures (PCs etc)Specific Neuro-HardwareDedicated circuits

    Aydn Ula

  • Generic architecturesConventional microprocessorsIntel Pentium, Power PC, etc AdvantagesHigh performances (clock frequency, etc)CheapSoftware environment available (NN tools, etc)DrawbacksToo generic, not optimized for very fast neural computations

    Aydn Ula

  • Classification of HardwareNN HardwareNeurochipsSpecial PurposeGeneral Purpose (Ni1000, L - Neuro)NeuroComputersSpecial Purpose (CNAPS, Synapse)General Purpose

    Aydn Ula

  • Specific Neuro-hardware circuitsCommercial chips CNAPS, Synapse, etc.AdvantagesCloser to the neural applicationsHigh performances in terms of speedDrawbacksNot optimized to specific applicationsAvailabilityDevelopment tools

    Aydn Ula

  • CNAPSSIMDOne instruction sequencing and control unitProcessor nodes (PN)Single dimensional array (only right or left nodes)

    Aydn Ula

  • CNAPS 1064

    Aydn Ula

  • CNAPS

    Aydn Ula

  • Dedicated circuitsA system where the functionality is buried in the hardware. For specific applications only not changeableAdvantagesOptimized for a specific applicationHigher performances than the other systemsDrawbacksHigh development costs in terms of time and money

    Aydn Ula

  • What type of hardware to be used in dedicated circuits ?Custom circuitsASIC (Application-Specific Integrated Circuit)Necessity to have good knowledge of the hardware designFixed architecture, hardly changeableOften expensive Programmable logicValuable to implement real time systemsFlexibilityLow development costsLower performances compared to ASIC (Frequency, etc.)

    Aydn Ula

  • Programmable logicField Programmable Gate Arrays (FPGAs)Matrix of logic cells Programmable interconnectionAdditional features (internal memories + embedded resources like multipliers, etc.)ReconfigurabilityWe can change the configurations as many times as desired

    Aydn Ula

  • Real Time SystemsExecution of applications with time constraints.Hard real-time systemsDigital fly-by-wire control system of an aircraft: No lateness is accepted. The lives of people depend on the correct working of the control system of the aircraft.Soft real-time systemsVending machine: Accept lower performance for lateness, it is not catastrophic when deadlines are not met. It will take longer to handle one client with the vending machine.

    Aydn Ula

  • Real Time Systemsms scale real time systemConnectionist retina for image processingArtificial Retina: combining an image sensor with a parallel architectures scale real time systemLevel 1 trigger in a HEP experiment

    Aydn Ula

  • Connectionist RetinaIntegration of a neural network in an artificial retinaScreenMatrix of Active Pixel sensorsCAN 8 bits ADC converter 256 levels of greyProcessing ArchitectureParallel system where neural networks are implemented

    ProcessingArchitecture

    Aydn Ula

  • Maharadja Processing ArchitectureMicro-controllerGeneric architecture executing sequential cost with low power consumptionMemory 256 Kbytes shared between processor, PEs, inputStore the network parametersUNE (Unit Neural SIMD Completely pipelined 16 bit internal data bus)Processors to compute the neurons outputsCommand bus manages all different operators in UNEInput/Output moduleData acquisition and storage of intermediate results

    Micro-controllerSequencer Command busInput/OutputunitInstruction BusUNE-0UNE-1UNE-2UNE-3MMMM

    Aydn Ula

  • Level 1 trigger in a HEP experimentHigh Energy Physics (Particle Physics)Neural networks have provided interesting results as triggers in HEP.Level 2 : H1 experiment 10 20 s Level 1 : Dirac experiment 2 s Particle RecognitionHigh timing constraints (in terms of latency and data throughput)

    Aydn Ula

  • Neural Network architecture....64128Execution time : ~500 ns

    Weights coded in 16 bitsStates coded in 8 bits with data arriving every BC=25ns4Electrons, tau, hadrons, jets

    Aydn Ula

  • Very Fast ArchitectureTanHPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPETanHTanHTanHACCACCACCACC256 PEsMatrix of n*m matrix elementsControl unitI/O moduleTanH are stored in LUTs1 matrix row computes a neuronThe results is back-propagated to calculate the output layer

    I/O moduleControl unit

    Aydn Ula

  • PE architectureXAccumulatorMultiplierWeights memInput data816Addr gen +Data incmd busControl ModuleData out

    Aydn Ula

  • Neuro-hardware todayGeneric Real time applicationsMicroprocessors technology (PCs, computers, i.e. software) is sufficient to implement most of neural applications in real-time (ms or sometimes s scale) This solution is cheapVery easy to manageConstrained Real time applicationsIt still remains specific applications where powerful computations are needed e.g. particle physicsIt still remains applications where other constraints have to be taken into consideration (Consumption, proximity of sensors, mixed integration, etc.)

    Aydn Ula

  • ClusteringIdea : Combine performances of different processors to perform massive parallel computations

    High speedconnection

    Aydn Ula

  • ClusteringAdvantagesTake advantage of the implicit parallelism of neural networksUtilization of systems already available (university, Labs, offices, etc.)High performances : Faster training of a neural net Very cheap compare to dedicated hardware

    Aydn Ula

  • ClusteringDrawbacksCommunications load : Need of very fast links between computers Software environment for parallel processingNot possible for embedded applications

    Aydn Ula

  • Hardware ImplementationsMost real-time applications do not need dedicated hardware implementationConventional architectures are generally appropriateClustering of generic architectures to combine performancesSome specific applications require other solutionsStrong Timing constraintsTechnology permits to utilize FPGAs FlexibilityMassive parallelism possibleOther constraints (consumption, etc.)Custom or programmable circuits

    Aydn Ula

  • Questions?

    Aydn Ula


74.419 Artificial Intelligence 2004 - Neural Networks - Neural Networks (NN) basic processing units general network architectures learning qualities and

Analysis and Optimization of Convolutional Neural Network ... · Convolutional Neural Network Architectures Master Thesis of Martin Thoma Department of Computer Science Institute

An Overview of Convolutional Neural Network Architectures ... · An Overview of Convolutional Neural Network Architectures for Deep Learning John Murphy 1 Microwa,y Inc. Fall 2016

Introduction to Neural Networks - GitHub Pages · Network Model • We focused on one example neural network, but one can also build neural networks with other architectures (meaning

Evolving Custom Convolutional Neural Network Architectures ...people.southwestern.edu/~schrum2/SCOPE/...SZTetris.pdf · architectures. Various custom architectures are tested in the

Convolutional Neural Networks: A Comprehensive Survey · Convolutional Neural Network, Network Architectures, VGGNet, ResNet, DenseNet. 1. INTRODUCTION Deep Learning is a broader

Neural Network Architectures and Learningmistis.inrialpes.fr/docs/biblioPlaneto/Tutorial...Neural Network Architectures and Learning Bogdan M. Wilamowski, Fellow Member, IEEE Auburn

MicroNets: Neural Network Architectures for Deploying

Neural Network Architectures for Prepositional Phrase ... · PDF fileNeural Network Architectures for Prepositional Phrase Attachment ... for Prepositional Phrase Attachment Disambiguation

Evolving Neural Network Architectures in a Computational Ecology

DEEP NEURAL NETWORK ARCHITECTURES FOR ...web.ics.purdue.edu/~elgamala/Xiaoyu.pdfM.S., Purdue University, May 2018. Deep Neural Network Architectures for Modu-lation Classi cation

Neural Network Programming with Javathe-eye.eu/public/Site-Dumps/r0bin0705_8fzydz_share.nxtcloud.net/mirror/New/A.I. by...Learning about neural network architectures 8 Monolayer networks

2tpy1wu Model Compression IIIT Hyderabad Girish …Learning Neural Network Architectures using Backpropagation Suraj Srinivas , R. Venkatesh Babu Loss Function Learning Neural Network

Investigation of Recurrent Neural Network architectures

DESIGNING NEURAL NETWORK ARCHITECTURES USING REINFORCEMENT

Boolean Weightless Neural Network Architecturesclok.uclan.ac.uk/5325/3/Armstrong James Final e-Thesis (Master Co… · Boolean Weightless Neural Network Architectures BY James Robert

Neural Network Hopfield model Kim, Il Joong. Contents Neural network: Introduction Definition & Application Network architectures Learning processes

a Survey Of Neuromorphic Computing And Neural - arXiv · A Survey of Neuromorphic Computing and Neural ... (VLSI) with analog ... neural network architectures themselves or for training,

Fully Binary Neural Network Model and Optimized Hardware ...ziyang.eecs.umich.edu/iesr/papers/coussy15apr.pdfFully Binary Neural Network Model and Optimized Hardware Architectures

On Using Deep Convolutional Neural Network Architectures ...dro.dur.ac.uk/24201/1/24201.pdf · On Using Deep Convolutional Neural Network Architectures for Object Classification

Automatic Generation of Neural Network Architectures Using a … · 18 Automatic Generation of Neural Network Architectures Using a Genetic Algorithm | 27.09.2016 Implemented an algorithm

Convolutional Neural Network Architectures: from …slazebni.cs.illinois.edu/spring17/lec01_cnn...Convolutional Neural Network Architectures: from LeNet to ResNet Lana Lazebnik Figure

DEEP NEURAL NETWORK ARCHITECTURES FOR MODULATION ...elgamala/Xiaoyu.pdf · xi ABSTRACT M.S., Purdue University, May 2018. Deep Neural Network Architectures for Modu-lation Classi

CommNets: Communicating Neural Network Architectures for

Discovering Gated Recurrent Neural Network Architecturesnn.cs.utexas.edu/downloads/papers/AdityaRawalThesis.pdfDiscovering Gated Recurrent Neural Network Architectures by Aditya Rawal,

Neural Network Architectures Aydın Ulaş 02 December 2004 [email protected]

Neural Autoregressive Distribution EstimationWe present Neural Autoregressive Distribution Estimation (NADE) models, which are neural network architectures applied to the problem of

Forex Exchange Rate Forecasting Using Deep Recurrent ... · Forex Exchange Rate Forecasting Using Deep Recurrent Neural Networks 3 2 Neural Network Architectures 2.1 Feedforward Neural

Model and Optimized Fully Binary Neural Network Hardware ...ziyang.eecs.umich.edu/iesr/lectures/coussy15apr-present.pdfFully Binary Neural Network Model and Optimized Hardware Architectures

Neural Network Architectures for Artificial Noses

Neural Architectures for Learning to Answer Questions · Neural Architectures for Learning to Answer Questions ... experienced players in the quiz show ... a neural network model

Neurmorphic Architectures...Artificial Neural Network Chips yEarly neuromorphic architectures were artificial neural network chips yExamples: yETANN : (1989) Entirely analog chip that

CSI NN: Reverse Engineering of Neural Network …CSI NN: Reverse Engineering of Neural Network Architectures Through Electromagnetic Side Channel Lejla Batina Radboud University, The

Dataflow Architectures: Flexible Platforms for Neural Network Simulation · 2014-11-12 · Dataflow Architectures: Flexible Platforms for Neural Network Simulation 821 3 NEURAL NETWORK

Neural Networksmgormley/courses/10601/slides/lecture12... · 2020-04-13 · Neural Network Architectures Even for a basic Neural Network, there are many design decisions to make: