Upload File

cs 4803 / 7643: deep learning...should match training data regularization: prevent the model from...

  • Home
  • Documents
  • CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:
49
CS 4803 / 7643: Deep Learning Dhruv Batra Georgia Tech Topics: – Optimization – Computing Gradients

Upload: others

Post on 12-Mar-2021

4 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

CS 4803 / 7643: Deep Learning

Dhruv Batra

Georgia Tech

Topics: – Optimization– Computing Gradients

Page 2: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Administrativia• HW1 Reminder

– Due: 09/09, 11:59pm

(C) Dhruv Batra 2

Page 3: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Recap from last time

(C) Dhruv Batra 3

Page 4: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Regularization

4

Data loss: Model predictions should match training data

Regularization: Prevent the model from doing too well on training data

= regularization strength(hyperparameter)

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Occam’s Razor: “Among competing hypotheses, the simplest is the best”William of Ockham, 1285 - 1347

Page 5: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Regularization

5

Data loss: Model predictions should match training data

Regularization: Prevent the model from doing too well on training data

= regularization strength(hyperparameter)

Simple examplesL2 regularization: L1 regularization: Elastic net (L1 + L2):

More complex:DropoutBatch normalizationStochastic depth, fractional pooling, etc

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 6: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

So far: Linear Classifiers

6

f(x) = Wx

Class scores

Page 7: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Hard cases for a linear classifier

7

Class 1: First and third quadrants

Class 2: Second and fourth quadrants

Class 1: 1 <= L2 norm <= 2

Class 2:Everything else

Class 1: Three modes

Class 2:Everything else

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 8: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Feature Extraction

Image features vs Neural Nets

8

f10 numbers giving scores for classes

training

training

10 numbers giving scores for classes

Krizhevsky, Sutskever, and Hinton, “Imagenet classification with deep convolutional neural networks”, NIPS 2012.Figure copyright Krizhevsky, Sutskever, and Hinton, 2012. Reproduced with permission.

Page 9: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

(Before) Linear score function:

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Neural networks: without the brain stuff

Page 10: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

10

(Before) Linear score function:

(Now) 2-layer Neural Network

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Neural networks: without the brain stuff

Page 11: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

11

x hW1 sW2

3072 100 10

Neural networks: without the brain stuff

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

(Before) Linear score function:

(Now) 2-layer Neural Network

Page 12: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

12

Neural networks: without the brain stuff

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

x hW1 sW2

3072 100 10

(Before) Linear score function:

(Now) 2-layer Neural Network

Page 13: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

13

(Before) Linear score function:

(Now) 2-layer Neural Network or 3-layer Neural Network

Neural networks: without the brain stuff

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 14: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Multilayer Networks• Cascaded “neurons”• The output from one layer is the input to the next• Each layer has its own sets of weights

(C) Dhruv Batra 14Image Credit: Andrej Karpathy, CS231n

Page 15: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

15

Impulses carried toward cell body

Impulses carried away from cell body

This image by Felipe Peruchois licensed under CC-BY 3.0

dendrite

cell body

axon

presynaptic terminal

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

https://thenounproject.com/term/neuron/214105/
https://creativecommons.org/licenses/by/3.0/us/
Page 16: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Sigmoid

tanh

ReLU

Leaky ReLU

Maxout

ELU

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Activation functions

Page 17: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Plan for Today• Optimization• Computing Gradients

(C) Dhruv Batra 17

Page 18: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Optimization

Page 19: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Supervised Learning• Input: x (images, text, emails…)• Output: y (spam or non-spam…)

• (Unknown) Target Function– f: X Y (the “true” mapping / reality)

• Data – (x1,y1), (x2,y2), …, (xN,yN)

• Model / Hypothesis Class– {h: X Y}– e.g. y = h(x) = sign(wTx)

• Loss Function– How good is a model wrt my data D?

• Learning = Search in hypothesis space– Find best h in model class.

(C) Dhruv Batra 19

Page 20: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Demo Time• https://playground.tensorflow.org

https://playground.tensorflow.org/
Page 21: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Strategy: Follow the slope

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 22: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

What is slope?• In 1-dimension, the derivative of a function:

23

Page 23: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

What is slope?• In d-dimension, recall partial derivatives:

24

Page 24: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

What is slope?• The gradient is the vector of (partial derivatives)

along each dimension

• Properties– The direction of steepest descent is the negative gradient– The slope in any direction is the dot product of the direction

with the gradient

25

Page 25: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

original W

negative gradient directionw1

w2

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

http://demonstrations.wolfram.com/VisualizingTheGradientVector/

Page 26: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Gradient Descent

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

http://demonstrations.wolfram.com/VisualizingTheGradientVector/
Page 27: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 28: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Full sum expensive when N is large!

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Gradient Descent has a problem

Page 29: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Full sum expensive when N is large!

Approximate sum using a minibatch of examples32 / 64 / 128 common

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Stochastic Gradient Descent (SGD)

Page 30: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Stochastic Gradient Descent (SGD)

Page 31: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Stochastic Gradient Descent (SGD)

Page 32: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Full sum expensive when N is large!

Approximate sum using a minibatch of examples32 / 64 / 128 common

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Stochastic Gradient Descent (SGD)

Page 33: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

(C) Dhruv Batra 34

Page 34: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

How do we compute gradients?• Analytic or “Manual” Differentiation

• Symbolic Differentiation

• Numerical Differentiation

• Automatic Differentiation– Forward mode AD– Reverse mode AD

• aka “backprop”

(C) Dhruv Batra 35

Page 35: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

(C) Dhruv Batra 36Figure Credit: Baydin et al. https://arxiv.org/abs/1502.05767

Page 36: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

(C) Dhruv Batra 37Figure Credit: Baydin et al. https://arxiv.org/abs/1502.05767

Page 37: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

(C) Dhruv Batra 38Figure Credit: Baydin et al. https://arxiv.org/abs/1502.05767

Page 38: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

(C) Dhruv Batra 39Figure Credit: Baydin et al. https://arxiv.org/abs/1502.05767

Page 39: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

How do we compute gradients?• Analytic or “Manual” Differentiation

• Symbolic Differentiation

• Numerical Differentiation

• Automatic Differentiation– Forward mode AD– Reverse mode AD

• aka “backprop”

(C) Dhruv Batra 40

Page 40: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

(C) Dhruv Batra 41By Brnbrnz (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)]

Page 41: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

gradient dW:

[?,?,?,?,?,?,?,?,?,…]

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 42: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

gradient dW:

[?,?,?,?,?,?,?,?,?,…]

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

W + h (first dim):

[0.34 + 0.0001,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25322

Page 43: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

gradient dW:

[-2.5,?,?,?,?,?,?,?,?,…]

(1.25322 - 1.25347)/0.0001= -2.5

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

W + h (first dim):

[0.34 + 0.0001,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25322

Page 44: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

gradient dW:

[-2.5,?,?,?,?,?,?,?,?,…]

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

W + h (second dim):

[0.34,-1.11 + 0.0001,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25353

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 45: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

gradient dW:

[-2.5,0.6,?,?,?,?,?,?,?,…]

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

W + h (second dim):

[0.34,-1.11 + 0.0001,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25353

(1.25353 - 1.25347)/0.0001= 0.6

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 46: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

gradient dW:

[-2.5,0.6,?,?,?,?,?,?,?,…]

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

W + h (third dim):

[0.34,-1.11,0.78 + 0.0001,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 47: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

gradient dW:

[-2.5,0.6,0,?,?,?,?,?,?,…]

current W:

[0.34,-1.11,0.78,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

W + h (third dim):

[0.34,-1.11,0.78 + 0.0001,0.12,0.55,2.81,-3.1,-1.5,0.33,…] loss 1.25347

(1.25347 - 1.25347)/0.0001= 0

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Page 48: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

Numerical gradient: slow :(, approximate :(, easy to write :)Analytic gradient: fast :), exact :), error-prone :(

In practice: Derive analytic gradient, check your implementation with numerical gradient. This is called a gradient check.

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Numerical vs Analytic Gradients

Page 49: CS 4803 / 7643: Deep Learning...should match training data Regularization: Prevent the model from doing too well on training data = regularization strength (hyperparameter) Slide Credit:

How do we compute gradients?• Analytic or “Manual” Differentiation

• Symbolic Differentiation

• Numerical Differentiation

• Automatic Differentiation– Forward mode AD– Reverse mode AD

• aka “backprop”

(C) Dhruv Batra 50


Gaussian Processes for Text Regressionetheses.whiterose.ac.uk/17619/1/thesis.pdf · 3.2 String kernel hyperparameter optimisation results. For each hyperparameter its original value

Open-Source Asynchronous Distributed Hyperparameter ... · Open-Source Asynchronous Distributed Hyperparameter Optimization Based on Apache Spark FOSDEM’20 02.02.2020. ... Inner

Provenance Supporting Hyperparameter Analysis in Deep

Speeding up Automatic Hyperparameter Optimization of Deep ...aad.informatik.uni-freiburg.de/...Extrapolation_of_Learning_Curves.pdf · Speeding up Automatic Hyperparameter Optimization

Hyperparameter Optimization via Sequential Uniform Designs

Predictive Process Monitoring with Hyperparameter Optimization

Mini-Course 6: Hyperparameter Optimization { Harmonica · Mini-Course 6: Hyperparameter Optimization { Harmonica Yang Yuan Computer Science Department Cornell University. ... the

Regularization, Ridge Regression - University of …courses.cs.washington.edu/.../regularization-xvalidation-lasso.pdf · Ridge Regression: Effect of Regularization 14 ! Solution

Bilevel Programming for Hyperparameter Optimization and

Hyperparameter optimization with approximate gradient

Sequence Modelling and Recurrent Neural Networks (RNNs) · •Neural Networks (recap) and Deep Learning •Improving DNN: Hyperparameter tuning, regularization, optimization •Convolutional

Bayesian Hyperparameter Optimization : Overfitting ... · Bayesian Hyperparameter Optimization: Overfitting, Ensembles and Conditional Spaces Thèse Julien-Charles Lévesque Sous

Performance Benchmarking of Parallel Hyperparameter Tuning ...gobbert/papers/Barajas_BDR2020.pdf · Performance Benchmarking of Parallel Hyperparameter Tuning for Deep Learning based

AGS FTC Team #7643 Cascade Effect

Bayesian Hyperparameter Optimization of Gaze Estimation ... · Bayesian Hyperparameter Optimization of Gaze Estimation Neural Networks Master’s Thesis Espoo, November 2, ... This

7643 presentación bbva pensiones españa 1

REDUCING THE SEARCH SPACE FOR HYPERPARAMETER OPTIMIZATION ...home.engineering.iastate.edu/~chinmay/files/papers/pgsrICASSP19.… · of hyperparameter optimization (HPO) addresses

Speeding up Automatic Hyperparameter Optimization of Deep

Hyperparameter Optimization 101

Massively Parallel Hyperparameter Tuning - arXivMassively Parallel Hyperparameter Tuning Liam Li 1, Kevin Jamieson2, Afshin Rostamizadeh3, Ekaterina Gonina3, Moritz Hardt 4, Benjamin

Bilevel Programming for Hyperparameter …proceedings.mlr.press/v80/franceschi18a/franceschi18a.pdfBilevel Programming for Hyperparameter Optimization and Meta-Learning Table 1. Links

HYPERNOMAD: Hyperparameter optimization of deep neural … · 2020. 10. 6. · BBO MADS and categorical variables HPO Computational experiments Discussion HYPERNOMAD: Hyperparameter

Automatisierte Hyperparameter Optimierung im Maschinellen ... · Automatisierte Hyperparameter Optimierung im Maschinellen Lernen Einfuhrung in Maschinelles Lernen Ziel des Maschinellen

Weight-Sharing for Hyperparameter Optimization in ...mkhodak/docs/FL2020Workshop.pdf · Weight-Sharing for Hyperparameter Optimization in Federated Learning Mikhail Khodak 1Tian Li

Hyperparameter Optimization - Sven Hafeneger

CHOPT : Automated Hyperparameter Optimization Framework ... · CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms We introduce an implemented

Convolution Neural Network Hyperparameter Optimization

Consola ST33xx-68 7643-000 Rivo M

Leveraging deep learning hyperparameter tuning frameworks · Leveraging deep learning hyperparameter tuning frameworks Derek Jensen, Donald Lucas, Cli˙ord Anderson-Bergman, Sonia

AutoNE: Hyperparameter Optimization for Massive …AutoNE: Hyperparameter Optimization for Massive Network Embedding Ke Tu∗ Tsinghua University [email protected] Jianxin

HYPERPARAMETER OPTIMIZATION OF DEEP CONVOLUTIONAL …

CS 4803 / 7643: Deep Learning

Speeding up Automatic Hyperparameter …ml.informatik.uni-freiburg.de/papers/15-IJCAI...Speeding up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of

Neural Network Hyperparameter Optimization · does the complexity of the space of possible hyperparameter configurations. Common characteristics of hyperparameter search spaces include

Hyperparameter Estimation in Exponential Family State ... · Hyperparameter Estimation in Exponential Family State Space Models Sonderforschungsbereich 386, Paper 6 (1995) ... tial