Upload File

7 neural networks learning

prev
next
out of 33
Download 7 Neural Networks Learning

Upload: ramon-lins

Post on 03-Apr-2018

227 views

Category:

Documents


0 download

Report

TRANSCRIPT

  • 7/28/2019 7 Neural Networks Learning

    1/33

    NeuralNetwo

    Learning

    Costfunc5

    MachineLearning

  • 7/28/2019 7 Neural Networks Learning

    2/33

    NeuralNetwork(Classifica2on)

    Binaryclassifica5on

    1outputunit

    Layer1 Layer2 Layer3 Layer4

    Mul5-classclassifica5

    Koutputunits

    totalno.oflayersinnetw

    no.ofunits(notcoun5nglayer

    pedestriancarm

    E.g.,,

  • 7/28/2019 7 Neural Networks Learning

    3/33

  • 7/28/2019 7 Neural Networks Learning

    4/33

    NeuralNetwo

    Learning

    Backpropaga

    algorithm

    MachineLearning

  • 7/28/2019 7 Neural Networks Learning

    5/33

    Gradientcomputa2on

    Needcodetocompute:

    - -

  • 7/28/2019 7 Neural Networks Learning

    6/33

    Gradientcomputa2on

    Givenonetrainingexample(,):

    Forwardpropaga5on:

    Layer1 Layer2 Layer

  • 7/28/2019 7 Neural Networks Learning

    7/33

    Gradientcomputa2on:Backpropaga2onalgorithm

    Intui5on:errorofnodeinlayer.

    Layer1 Layer2 Lay

    Foreachoutputunit(layerL=4)

  • 7/28/2019 7 Neural Networks Learning

    8/33

    Backpropaga2onalgorithmTrainingset

    Set(forall).

    For

    Set

    Performforwardpropaga5ontocomputefor

    Using,compute

    Compute

  • 7/28/2019 7 Neural Networks Learning

    9/33

    NeuralNetwo

    Learning

    Backpropaga

    intui5on

    MachineLearning

  • 7/28/2019 7 Neural Networks Learning

    10/33

    ForwardPropaga2on

  • 7/28/2019 7 Neural Networks Learning

    11/33

    ForwardPropaga2on

  • 7/28/2019 7 Neural Networks Learning

    12/33

    Whatisbackpropaga2ondoing?

    Focusingonasingleexample,,thecaseof1outp

    andignoringregulariza5on(),

    (Thinkof)

    I.e.howwellisthenetworkdoingonexamplei?

  • 7/28/2019 7 Neural Networks Learning

    13/33

    ForwardPropaga2on

    errorofcostfor(unitinlayer).

    Formally, (for),where

  • 7/28/2019 7 Neural Networks Learning

    14/33

    NeuralNetwo

    LearningImplementa5on

    note:Unrolling

    parameters

    MachineLearning

  • 7/28/2019 7 Neural Networks Learning

    15/33

    Advancedop2miza2on

    function [jVal, gradient] = costFunction(thet

    optTheta = fminunc(@costFunction, initialThet

    NeuralNetwork(L=4):

    -matrices(Theta1, Theta2, Th

    -matrices(D1, D2, D3)

    Unrollintovectors

  • 7/28/2019 7 Neural Networks Learning

    16/33

    Example

    thetaVec = [ Theta1(:); Theta2(:); Theta3(:)]DVec = [D1(:); D2(:); D3(:)];

    Theta1 = reshape(thetaVec(1:110),10,11);Theta2 = reshape(thetaVec(111:220),10,11);Theta3 = reshape(thetaVec(221:231),1,11);

  • 7/28/2019 7 Neural Networks Learning

    17/33

    Haveini5alparameters.

    UnrolltogetinitialThetatopasstofminunc(@costFunction, initialTheta, options)

    LearningAlgorithm

    function [jval, gradientVec] = costFunction(t

    FromthetaVec, get.

    Useforwardprop/backproptocomputeand.

    UnrolltogetgradientVec.

  • 7/28/2019 7 Neural Networks Learning

    18/33

    NeuralNetwo

    Learning

    Gradientche

    MachineLearning

  • 7/28/2019 7 Neural Networks Learning

    19/33

    Numericales2ma2onofgradients

    Implement:gradApprox = (J(theta + EPSILON) J(theta

    EPSILON)) /(2*EPSILON)

  • 7/28/2019 7 Neural Networks Learning

    20/33

    Parametervector

    (E.g.isunrolledversionof

  • 7/28/2019 7 Neural Networks Learning

    21/33

    for i = 1:n,thetaPlus = theta;thetaPlus(i) = thetaPlus(i) + EPSILON;

    thetaMinus = theta;thetaMinus(i) = thetaMinus(i) EPSILON;gradApprox(i) = (J(thetaPlus) J(thetaMin

    /(2*EPSILON);end;

    CheckthatgradApproxDVec

  • 7/28/2019 7 Neural Networks Learning

    22/33

    Implementa2onNote:

    - ImplementbackproptocomputeDVec(unrolled- Implementnumericalgradientchecktocomputegra- Makesuretheygivesimilarvalues.- Turnoffgradientchecking.Usingbackpropcodefor

    Important:

    - Besuretodisableyourgradientcheckingcodebeforyourclassifier.Ifyourunnumericalgradientcomput

    everyitera5onofgradientdescent(orintheinnerlo

    costFunction())yourcodewillbeveryslow.

  • 7/28/2019 7 Neural Networks Learning

    23/33

    NeuralNetwo

    Learning

    Random

    ini5aliza5oMachineLearning

  • 7/28/2019 7 Neural Networks Learning

    24/33

    Ini2alvalueof

    Forgradientdescentandadvancedop5miza5o

    method,needini5alvaluefor.

    Considergradientdescent

    Set ?

    optTheta = fminunc(@costFunction,initialTheta, options)

    initialTheta = zeros(n,1)

  • 7/28/2019 7 Neural Networks Learning

    25/33

    Zeroini2aliza2on

    A_ereachupdate,parameterscorrespondingtoinputsgoinginto

    twohiddenunitsareiden5cal.

  • 7/28/2019 7 Neural Networks Learning

    26/33

    Randomini2aliza2on:Symmetrybreaking

    Ini5alizeeachtoarandomvaluein

    (i.e.)

    E.g.

    Theta1 = rand(10,11)*(2*INIT_EPSILON

    - INIT_EPSILON;

    Theta2 = rand(1,11)*(2*INIT_EPSILON- INIT_EPSILON;

  • 7/28/2019 7 Neural Networks Learning

    27/33

    NeuralNetwo

    Learning

    Pu`ngit

    togetherMachineLearning

  • 7/28/2019 7 Neural Networks Learning

    28/33

    Traininganeuralnetwork

    Pickanetworkarchitecture(connec5vitypaernbetweenneurons

    No.ofinputunits:Dimensionoffeatures

    No.outputunits:Numberofclasses

    Reasonabledefault:1hiddenlayer,orif>1hiddenlayer,havesam

    hiddenunitsineverylayer(usuallythemorethebeer)

  • 7/28/2019 7 Neural Networks Learning

    29/33

    Traininganeuralnetwork

    1. Randomlyini5alizeweights2. Implementforwardpropaga5ontogetforan3. Implementcodetocomputecostfunc5on4. Implementbackproptocomputepar5alderiva5vesfor i = 1:m

    Performforwardpropaga5onandbackpropaga5on

    example(Getac5va5onsanddeltatermsfor

  • 7/28/2019 7 Neural Networks Learning

    30/33

    Traininganeuralnetwork

    . Usegradientcheckingtocomparecomputbackpropaga5onvs.usingnumericales5mateofgra

    of.Thendisablegradientcheckingcode.

    6. Usegradientdescentoradvancedop5miza5onmetbackpropaga5ontotrytominimizeasafunc5o

    parameters

  • 7/28/2019 7 Neural Networks Learning

    31/33

  • 7/28/2019 7 Neural Networks Learning

    32/33

    NeuralNetwo

    Learning

    Backpropaga5on

    example:Autondriving(op5ona

    MachineLearning

  • 7/28/2019 7 Neural Networks Learning

    33/33

    [CourtesyofDeanPomerleau]


Learning Algorithm and Neural Networks

Lecture 10: Neural Networks and Deep Learningsaravanan-thirumuruganathan.github.io/cse5334... · Neural Networks Deep Learning Convolutional Neural Networks Recurrent Neural Networks

Neural Networks - cs.cmu.edubhiksha/courses/deep... · about neural networks (a.k.a deep learning) This guy learned about neural networks (a.k.a deep learning) Objectives of this

Modal Learning Neural Networks - WSEAS

Neural Networks and Deep Learning

Neural Networks and Deep Learning - CSE - IIT Kanpur€¦ · Neural Networks and Deep Learning . Example Learning Problem . Example Learning Problem ... –Radial basis functions

Deep Learning in Neural Networks: An Overviejuergen/DeepLearning28May2014.pdf · Deep Learning in Neural Networks: An Overview ... 1 Introduction to Deep Learning ... Neural Fitted

Machine Learning: Introduction to Neural Networks

Arti cal Neural Networks - University of Chicagottic.uchicago.edu/~shubhendu/Slides/NeuralNets1.pdf · Arti cal Neural Networks ... • Deep Learning: Convolutional Neural Networks

Perceptrons and Learning Learning in Neural Networks

Neural Networks - Lecture 81 Unsupervised competitive learning Particularities of unsupervised learning Data clustering Neural networks for clustering

Deep Learning & Artificial Intelligence · Schedule 17.10.2018 Introduction 24.10.2018 Basic Neural Networks 31.10.2018 Training Neural Networks 07.11.2018 Convolutional Neural Networks

Course 395: Machine Learning - Lectures...Stefanos Zafeiriou Machine Learning (395) Neural Networks Neural Networks The Perceptron (1958) Multilayer Neural Networks (50’s, 60’s)

Supervised Learning in Neural Networks

Machine Learning and Neural Networks

CS536: Machine Learning Artificial Neural Networks Neural Networks

Métodos Actuales de Machine Learning Neural Networksdc.exa.unrc.edu.ar/rio2016/sites/default/files/Neural Networks... · Métodos Actuales de Machine Learning Neural Networks

Artificial Neural Networks Lect3: Neural Network Learning rules

Multilayer neural networks: learning models and applicationsdeim.urv.cat/~sergio.gomez/papers/Gomez-Multilayer_neural_networks... · Multilayer neural networks: learning models and

Deep Learning in Neural Networks: An Overviewpeople.idsia.ch/~juergen/DeepLearning2July2014.pdf · Deep Learning in Neural Networks: An Overview ... numerous neural network experts

Deep Learning: What Makes Neural Networks Great Again?infochim.u-strasbg.fr/CS3_2018/Presentations/... · • Convolutional neural networks (CNN) • Recurrent neural networks (RNN)

DEEP LEARNING AND NEURAL NETWORKS Deep Learning History of Artificial Neural Networks Keras ... Recurrent Neural Networks (RNNs) Long Term Short Memory (I-ST M) ... Fractal Analytics

Unsupervised Neural Networks Learning Rules, BCM Learning Rule

Neural Networks for Machine Learning Lecture 3a Learning

Neural Networks and Deep Learning

Reinforcement Learning and Neural networks

Unsupervised learning Networks Associative Memory Networks ELE571 Digital Neural Networks

DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND …wcohen/10-605/deep-1.pdf · DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY 1. ... Convolutional Neural Networks for

Machine learning 2 - Neural Networks

Introduction to Deep Neural Networks - Deep Learning · Introduction to Deep Neural Networks 0. Logistics Spring 2020 1. Neural Networks are taking over! •Neural networks have become

Convolutional Neural Networks for Deep Learning: An Introduction · 2017-10-05 · Introduction: Convolutional Neural Networks (CNN) Convolutional Neural Networks: A deep learning

Learning Neural Networks (NN)

Neural Networks Self Organizing Mapscancelli/retineu11_12/SOM.pdf · Neural Networks Self Organizing Maps NN 4 1 Unsupervised Learning Neural networks for unsupervised learning attempt

Artificial Neural Networks Connectionist Learning Machinesci.louisville.edu/kerem/evraklar/sunum/CECS694_01_seminar.pdfArtificial Neural Networks Connectionist Learning Machines

Introduction to Neural Networks Supervised learning: neural networks ...vision.psych.umn.edu/users/kersten/kersten-lab/courses/Psy5038WF... · Introduction to Neural Networks Supervised