introduction to deep learning - nucleartalent.github.io€¦ · introduction to deep learning raghu...
TRANSCRIPT
Introduction to Deep Learning
Raghu RamanujanDept. of Mathematics and Computer Science
Davidson College
Neural Network Fundamentals
The Neuron
Image courtesy: http://bio3520.nicerweb.com/Locked/chap/ch03/neuron.html
The Neuron
Image courtesy: http://learn.genetics.utah.edu/content/addiction/neurons/
“Real” Neuron
Image courtesy: http://bio3520.nicerweb.com/Locked/chap/ch03/neuron.html
Artificial Neurons
x1
x2
∑ 𝜎
w1
w 2
y
Activation Function
Artificial Neurons
x1
x2
∑ 𝜎
w1
w 2
y
Artificial Neurons
x1
x2
N y
w1
w2
Fully-Connected Neural Network
x1
x2
N
xn
… N
… N y
Input Layer Hidden Layer Output Layer
Backpropagation
Backpropagation:
Initialize all weights in the network to small, random numbers.
loop
for each training example (x, y) do
ForwardProp:
For each hidden unit h, ah = �(neth) = �(P
i wihxi)
y = ak = �(netk) = �(P
h whah)
BackProp:
�k =@J
@netk= (y � y)y(1� y)
For each weight wh, wh wh � ⌘�kah
For each hidden unit h, �h = �kwhah(1� ah)
For each weight wih, wih wih � ⌘�hxi
end for
end loop
1
Modern Neural Networks
Automatic Differentiation
Automatic Differentiation
• Use the abstraction of a computational graph
• Define your computation and let engine worry about optimization
Computational Graph
*
x1
w1
p
5
3
15
Forward Pass• Apply the operator
Computational Graph
*
x1
w1
p
5
@p
@w1= x1
<latexit sha1_base64="4OcFFx0Jh2f/+t7y4y4KfBpPGho=">AAACDnicbZBNS8MwGMdTX+d8q3r0EhwDT6PVgV6EgRePE9wLrKOkWbqFpWlIUnWUfgIvfhUvHhTx6tmb38Z0K6ibfwj8+D/Pk+T5B4JRpR3ny1paXlldWy9tlDe3tnd27b39tooTiUkLxyyW3QApwignLU01I10hCYoCRjrB+DKvd26JVDTmN3oiSD9CQ05DipE2lm9XvVAinHoCSU0RgyL74TvfzeAFvPdd3644NWcquAhuARVQqOnbn94gxklEuMYMKdVzHaH7aX4xZiQre4kiAuExGpKeQY4iovrpdJ0MVo0zgGEszeEaTt3fEymKlJpEgemMkB6p+Vpu/lfrJTo876eUi0QTjmcPhQmDOoZ5NnBAJcGaTQwgLKn5K8QjZPLRJsGyCcGdX3kR2ic197TmXtcrjXoRRwkcgiNwDFxwBhrgCjRBC2DwAJ7AC3i1Hq1n6816n7UuWcXMAfgj6+MbrOmbyA==</latexit>
Backward Pass• Adjust parameter using local
gradient 3 (scaled by a learning rate)
1
3
15
Computational Graph
*
x1
w1
p
4.7
@p
@w1= x1
<latexit sha1_base64="4OcFFx0Jh2f/+t7y4y4KfBpPGho=">AAACDnicbZBNS8MwGMdTX+d8q3r0EhwDT6PVgV6EgRePE9wLrKOkWbqFpWlIUnWUfgIvfhUvHhTx6tmb38Z0K6ibfwj8+D/Pk+T5B4JRpR3ny1paXlldWy9tlDe3tnd27b39tooTiUkLxyyW3QApwignLU01I10hCYoCRjrB+DKvd26JVDTmN3oiSD9CQ05DipE2lm9XvVAinHoCSU0RgyL74TvfzeAFvPdd3644NWcquAhuARVQqOnbn94gxklEuMYMKdVzHaH7aX4xZiQre4kiAuExGpKeQY4iovrpdJ0MVo0zgGEszeEaTt3fEymKlJpEgemMkB6p+Vpu/lfrJTo876eUi0QTjmcPhQmDOoZ5NnBAJcGaTQwgLKn5K8QjZPLRJsGyCcGdX3kR2ic197TmXtcrjXoRRwkcgiNwDFxwBhrgCjRBC2DwAJ7AC3i1Hq1n6816n7UuWcXMAfgj6+MbrOmbyA==</latexit>
Backward Pass• Adjust parameter using local
gradient 3 (scaled by a learning rate)
Computational Graph
*
x1
w1
p
*
x2
w2
q
+r
@r
@p= 1
@p
@w1= x1
<latexit sha1_base64="6z1qJapV2I50rfHl5OdrCWLUiuo=">AAACOnicbVDLSgMxFM3UVx1fVZdugkVxVSZa0I1QcOOyBfuAThkyaaYNzcyEJKOWof6WG7/CnQs3LhRx6weYaQvW1gMXDueee5N7fMGZ0o7zYuWWlldW1/Lr9sbm1vZOYXevoeJEElonMY9ly8eKchbRumaa05aQFIc+p01/cJX1m7dUKhZHN3ooaCfEvYgFjGBtJK9QcwOJSeoKLDXDHMrRLxcjeHwJkevaD6bmjGLGeOehsfXeQ7btFYpOyRkDLhI0JUUwRdUrPLvdmCQhjTThWKk2coTupNluwunIdhNFBSYD3KNtQyMcUtVJx6eP4JFRujCIpalIw7E6O5HiUKlh6BtniHVfzfcy8b9eO9HBRSdlkUg0jcjkoSDhUMcwyxF2maRE86EhmEhm/gpJH5uItEk7CwHNn7xIGqcldFZCtXKxUp7GkQcH4BCcAATOQQVcgyqoAwIewSt4Bx/Wk/VmfVpfE2vOms7sgz+wvn8AAzisLA==</latexit>
@r
@w1=
@r
@p
@p
@w1<latexit sha1_base64="THrO4qcAq6idU8u9XsI/RMV3GxU=">AAACSXicfVDNS8MwHE03P+b8qnr0EhyCp9HqQC/CwIvHCe4D1lLTLN2CaRuSVBml/54Xb978H7x4UMST6VZQO/FB4PHe+/2SPJ8zKpVlPRuV6tLyymptrb6+sbm1be7s9mScCEy6OGaxGPhIEkYj0lVUMTLggqDQZ6Tv317kfv+OCEnj6FpNOXFDNI5oQDFSWvLMGycQCKcOR0JRxKDIvvm9Z2fwHP6T4FnZ5aX5umc2rKY1A1wkdkEaoEDHM5+cUYyTkEQKMyTl0La4ctN8J2YkqzuJJBzhWzQmQ00jFBLpprMmMniolREMYqFPpOBM/TmRolDKaejrZIjURJa9XPzLGyYqOHNTGvFEkQjPLwoSBlUM81rhiAqCFZtqgrCg+q0QT5CuRuny8xLs8pcXSe+4aZ807atWo90q6qiBfXAAjoANTkEbXIIO6AIMHsALeAPvxqPxanwYn/NoxShm9sAvVKpf8nu0CA==</latexit>
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L
yp
q
ŷ r
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L
yp
q
ŷ
3
2
4
-2
r
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L
yp=12
q=-4
ŷ
3
2
4
-2
r
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L
yp=12
q=-4
ŷ=8
3
2
4
-2
r
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L
yp=12
q=-4
ŷ=8
3
2
4
-2
r
6
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
@L
@L= 1
<latexit sha1_base64="XNl+GJYEUeI1bvNoQx18t/ji9EQ=">AAACC3icbZDLSgMxFIbP1Futt1GXbkKL4KrMaEE3QsGNCxcV7AU6pWTSTBuayQxJRihD9258FTcuFHHrC7jzbcy0s6itPwQ+/nNOkvP7MWdKO86PVVhb39jcKm6Xdnb39g/sw6OWihJJaJNEPJIdHyvKmaBNzTSnnVhSHPqctv3xTVZvP1KpWCQe9CSmvRAPBQsYwdpYfbvsBRKT1Iux1AxzdDddZHSN3FLfrjhVZya0Cm4OFcjV6Nvf3iAiSUiFJhwr1XWdWPfS7FbC6bTkJYrGmIzxkHYNChxS1Utnu0zRqXEGKIikOUKjmbs4keJQqUnom84Q65FarmXmf7VuooOrXspEnGgqyPyhIOFIRygLBg2YpETziQFMJDN/RWSETTjaxJeF4C6vvAqt86p7UXXva5V6LY+jCCdQhjNw4RLqcAsNaAKBJ3iBN3i3nq1X68P6nLcWrHzmGP7I+voFiNqZ/g==</latexit>
1
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
1
@L
@r= 2r
= �4<latexit sha1_base64="vr9YGXam8ErvNYTFRb6+4zQ0AAQ=">AAACGHicbZDLSsNAFIYn9VbjLerSzWBR3FiTWtCNUHDjwkUFe4EmlMl00g6dTMLMRCghj+HGV3HjQhG33fk2TtqAWv1h4OM/53Dm/H7MqFS2/WmUlpZXVtfK6+bG5tb2jrW715ZRIjBp4YhFousjSRjlpKWoYqQbC4JCn5GOP77O650HIiSN+L2axMQL0ZDTgGKktNW3ztxAIJy6MRKKIgZvs28WGTyGV7AmoOuaOZ3Wzb5Vsav2TPAvOAVUQKFm35q6gwgnIeEKMyRlz7Fj5aX5CsxIZrqJJDHCYzQkPY0chUR66eywDB5pZwCDSOjHFZy5PydSFEo5CX3dGSI1kou13Pyv1ktUcOmllMeJIhzPFwUJgyqCeUpwQAXBik00ICyo/ivEI6STUjrLPARn8eS/0K5VnfOqc1evNOpFHGVwAA7BCXDABWiAG9AELYDBI3gGr+DNeDJejHfjY95aMoqZffBLxvQLUuSdRQ==</latexit>
-4
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
1-4
@L
@y=
@L
@r
@r
@y
= �4 ·�1
= 4<latexit sha1_base64="0dbDqfA6J1Ip5Can/ze0KiXA79s=">AAACb3icfVHLSgMxFM2Mr1pftS5cVCRYFF20zGhBN0LBjQsXFewDOqVk0kwbmnmQ3BHKMFs/0J3/4MY/MNNW1Fq8EDicc+69yYkbCa7Ast4Mc2V1bX0jt5nf2t7Z3SvsF1sqjCVlTRqKUHZcopjgAWsCB8E6kWTEdwVru+O7TG8/M6l4GDzBJGI9nwwD7nFKQFP9wovjSUITJyISOBH4If3GzohAMklTfIZv8T8+mS6qctkUx8lngyo17NBBCLhif1G1fL9QtqrWtPBfYM9BGc2r0S+8OoOQxj4LgAqiVNe2Iugl2U4qWJp3YsUiQsdkyLoaBsRnqpdM80rxqWYG2AulPgHgKfuzIyG+UhPf1U6fwEgtahm5TOvG4N30Eh5EMbCAzhZ5scAQ4ix8POCSURATDQiVXN8V0xHR0YH+oiwEe/HJf0HrsmpfVe3HWrlem8eRQyV0gs6Rja5RHd2jBmoiit6NolEyjowP89A8NvHMahrzngP0q8yLT33iudE=</latexit>
4
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
1-44
@L
@p=
@L
@y
@y
@p
= 4 · 1= 4<latexit sha1_base64="izSwUm33oCSjjYTwL+SaUBlPRoU=">AAACbXicfVHLSgMxFM2Mr1pfo+LCBxIsPlZlRgu6EQpuXLioYGuhU0omzbShmQfJHaEMs/ML3fkLbvwFM219teKFwOGcc+9NTrxYcAW2/WqYc/MLi0uF5eLK6tr6hrW51VBRIimr00hEsukRxQQPWR04CNaMJSOBJ9ijN7jJ9ccnJhWPwgcYxqwdkF7IfU4JaKpjPbu+JDR1YyKBE4Hvsm8cZ/gEX+N/HG6fQDrMsmnPJ/9rlusW83EV7NJuBNj5Ioodq2SX7VHhWeBMQAlNqtaxXtxuRJOAhUAFUarl2DG003wXFSwruoliMaED0mMtDUMSMNVOR2ll+FgzXexHUp8Q8Ij92ZGSQKlh4GlnQKCvprWc/EtrJeBftVMexgmwkI4X+YnAEOE8etzlklEQQw0IlVzfFdM+0cGB/qA8BGf6ybOgcV52LsrOfaVUrUziKKB9dITOkIMuURXdohqqI4reDMvYNfaMd3PHPDAPx1bTmPRso19lnn4AOWa5Xw==</latexit>
4
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
1-44
4
@L
@w1=
@L
@p
@p
@w1
= 4x1
= 12<latexit sha1_base64="43qIgw0wGoy5GLpRFSGJko2AlrU=">AAACYnicfVFNS8MwGE7r16wf29xRD8GheBrNHOhFGHjx4GGC+4B1lDRLt2DahiRVR+mf9ObJiz/EdJuoU3wh8LzP834kTwLBmdKu+2rZa+sbm1ulbWdnd2+/XKke9FSSSkK7JOGJHARYUc5i2tVMczoQkuIo4LQfPFwXev+RSsWS+F7PBB1FeBKzkBGsDeVXZl4oMck8gaVmmMPb/As/+SiHp/AK/lMj8lVVrE7wPKcY0oLPPvpMUNPxK3W34c4D/gZoCepgGR2/8uKNE5JGNNaEY6WGyBV6lBWrCKe546WKCkwe8IQODYxxRNUom1uUwxPDjGGYSHNiDefs944MR0rNosBURlhP1apWkH9pw1SHl6OMxSLVNCaLRWHKoU5g4TccM0mJ5jMDMJHM3BWSKTaOafMrhQlo9cm/Qa/ZQOcNdNeqt1tLO0rgEByDM4DABWiDG9ABXUDAm7Vh7Vtl69127KpdW5Ta1rKnBn6EffQBj5Ozyw==</latexit>
12
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
4
-2
r=-2
6
1-44
412
w1 w1 � ↵@L
@w1<latexit sha1_base64="bki58x5au8fl2b+AEs0qq8boW3A=">AAACJXicbVDLSgMxFM3UV62vUZdugkVwY5nRgi5cFNy4cFHBPqBThjtppg3NPEgyljLMz7jxV9y4sIjgyl8x0xbU1gOBk3Puvck9XsyZVJb1aRRWVtfWN4qbpa3tnd09c/+gKaNEENogEY9E2wNJOQtpQzHFaTsWFAKP05Y3vMn91iMVkkXhgxrHtBtAP2Q+I6C05JrXI9fGDqe+AiGiEc6vZ9gBHg8AO74AkjoxCMWA47vsh+u6rOSaZatiTYGXiT0nZTRH3TUnTi8iSUBDRThI2bGtWHXTfCbhNCs5iaQxkCH0aUfTEAIqu+l0ywyfaKWH/UjoEyo8VX93pBBIOQ48XRmAGshFLxf/8zqJ8q+6KQvjRNGQzB7yE45VhPPIcI8JShQfawJEMP1XTAago1E62DwEe3HlZdI8r9gXFfu+Wq5V53EU0RE6RqfIRpeohm5RHTUQQU/oBb2hifFsvBrvxsestGDMew7RHxhf31AMpH4=</latexit>
w1 4� (0.1)(12) = 2.8<latexit sha1_base64="uS/vfz63Q8HzYhcQqdZNxowhdMs=">AAACDHicbVDLSgMxFM34rPVVdekmWIR24TCpBbsRCm5cVrAPaIeSSTNtaCYzJBlLGfoBbvwVNy4UcesHuPNvzLSz0NYDgcM553JzjxdxprTjfFtr6xubW9u5nfzu3v7BYeHouKXCWBLaJCEPZcfDinImaFMzzWknkhQHHqdtb3yT+u0HKhULxb2eRtQN8FAwnxGsjdQvFCd9BHuc+hpLGU5gFV7AkmOjcglVyvAaVuxa3qQc25kDrhKUkSLI0OgXvnqDkMQBFZpwrFQXOZF2Eyw1I5zO8r1Y0QiTMR7SrqECB1S5yfyYGTw3ygD6oTRPaDhXf08kOFBqGngmGWA9UsteKv7ndWPt19yEiSjWVJDFIj/mUIcwbQYOmKRE86khmEhm/grJCEtMtOkvLQEtn7xKWhUbXdrorlqsV7M6cuAUnIESQOAK1MEtaIAmIOARPINX8GY9WS/Wu/WxiK5Z2cwJ+APr8wecn5bL</latexit>
Computational Graph
*
x1
w1
*
x2
w2
+ - ^2L=4
yp=12
q=-4
ŷ=8
3
2
2.8
-2
r=-2
6
1-44
412
w1 w1 � ↵@L
@w1<latexit sha1_base64="bki58x5au8fl2b+AEs0qq8boW3A=">AAACJXicbVDLSgMxFM3UV62vUZdugkVwY5nRgi5cFNy4cFHBPqBThjtppg3NPEgyljLMz7jxV9y4sIjgyl8x0xbU1gOBk3Puvck9XsyZVJb1aRRWVtfWN4qbpa3tnd09c/+gKaNEENogEY9E2wNJOQtpQzHFaTsWFAKP05Y3vMn91iMVkkXhgxrHtBtAP2Q+I6C05JrXI9fGDqe+AiGiEc6vZ9gBHg8AO74AkjoxCMWA47vsh+u6rOSaZatiTYGXiT0nZTRH3TUnTi8iSUBDRThI2bGtWHXTfCbhNCs5iaQxkCH0aUfTEAIqu+l0ywyfaKWH/UjoEyo8VX93pBBIOQ48XRmAGshFLxf/8zqJ8q+6KQvjRNGQzB7yE45VhPPIcI8JShQfawJEMP1XTAago1E62DwEe3HlZdI8r9gXFfu+Wq5V53EU0RE6RqfIRpeohm5RHTUQQU/oBb2hifFsvBrvxsestGDMew7RHxhf31AMpH4=</latexit>
w1 4� (0.1)(12) = 2.8<latexit sha1_base64="uS/vfz63Q8HzYhcQqdZNxowhdMs=">AAACDHicbVDLSgMxFM34rPVVdekmWIR24TCpBbsRCm5cVrAPaIeSSTNtaCYzJBlLGfoBbvwVNy4UcesHuPNvzLSz0NYDgcM553JzjxdxprTjfFtr6xubW9u5nfzu3v7BYeHouKXCWBLaJCEPZcfDinImaFMzzWknkhQHHqdtb3yT+u0HKhULxb2eRtQN8FAwnxGsjdQvFCd9BHuc+hpLGU5gFV7AkmOjcglVyvAaVuxa3qQc25kDrhKUkSLI0OgXvnqDkMQBFZpwrFQXOZF2Eyw1I5zO8r1Y0QiTMR7SrqECB1S5yfyYGTw3ygD6oTRPaDhXf08kOFBqGngmGWA9UsteKv7ndWPt19yEiSjWVJDFIj/mUIcwbQYOmKRE86khmEhm/grJCEtMtOkvLQEtn7xKWhUbXdrorlqsV7M6cuAUnIESQOAK1MEtaIAmIOARPINX8GY9WS/Wu/WxiK5Z2cwJ+APr8wecn5bL</latexit>
Dynamics of Learning in Deep Networks
Choosing an Activation Function
f(x) =1
1 + e�x<latexit sha1_base64="pAuULLasilgIl49Yya7mMtX+s3o=">AAACAnicbVDLSgMxFM3UV62vUVfiJliEilgmKtiNUHDjsoJ9QDuWTJqpoZnMkGSkZRjc+CtuXCji1q9w59+YtrPQ1gMXDufcy733eBFnSjvOt5VbWFxaXsmvFtbWNza37O2dhgpjSWidhDyULQ8rypmgdc00p61IUhx4nDa9wdXYbz5QqVgobvUoom6A+4L5jGBtpK6955eGR/ASdnyJSYLSBB3Tu+RkmKZdu+iUnQngPEEZKYIMta791emFJA6o0IRjpdrIibSbYKkZ4TQtdGJFI0wGuE/bhgocUOUmkxdSeGiUHvRDaUpoOFF/TyQ4UGoUeKYzwPpezXpj8T+vHWu/4iZMRLGmgkwX+TGHOoTjPGCPSUo0HxmCiWTmVkjusQlDm9QKJgQ0+/I8aZyW0VkZ3ZwXq5UsjjzYBwegBBC4AFVwDWqgDgh4BM/gFbxZT9aL9W59TFtzVjazC/7A+vwBYMGWHw==</latexit>
f(x) =ex � e�x
ex + e�x<latexit sha1_base64="VlPMneKQJ07Q1ZffkhwbOne9y9s=">AAACEXicbVDLSgMxFM34rPVVdekmWISKtMyoYDdCwY3LCvYB7bRk0jttaOZBkpGWob/gxl9x40IRt+7c+Tem0xG09UDg3HPu5eYeJ+RMKtP8MpaWV1bX1jMb2c2t7Z3d3N5+XQaRoFCjAQ9E0yESOPOhppji0AwFEM/h0HCG11O/cQ9CssC/U+MQbI/0feYySpSWurmCWxid4CvcdgWhMXRGuIihExdHk0lSnf5U3VzeLJkJ8CKxUpJHKard3Ge7F9DIA19RTqRsWWao7JgIxSiHSbYdSQgJHZI+tDT1iQfSjpOLJvhYKz3sBkI/X+FE/T0RE0/KsefoTo+ogZz3puJ/XitSbtmOmR9GCnw6W+RGHKsAT+PBPSaAKj7WhFDB9F8xHRCdjdIhZnUI1vzJi6R+VrLOS9btRb5STuPIoEN0hArIQpeogm5QFdUQRQ/oCb2gV+PReDbejPdZ65KRzhygPzA+vgH1O5vW</latexit>
f(x) = max(0, x)<latexit sha1_base64="OCe/em5zuw6TQA53zhugZhrtMac=">AAAB+nicbVBNS8NAEJ3Ur1q/Uj16WSxCC1ISFexFKHjxWMF+QBvKZrtpl+4mYXejLbU/xYsHRbz6S7z5b9y2OWjrg4HHezPMzPNjzpR2nG8rs7a+sbmV3c7t7O7tH9j5w4aKEklonUQ8ki0fK8pZSOuaaU5bsaRY+Jw2/eHNzG8+UKlYFN7rcUw9gfshCxjB2khdOx8URyV0jToCj4rOGRqVunbBKTtzoFXipqQAKWpd+6vTi0giaKgJx0q1XSfW3gRLzQin01wnUTTGZIj7tG1oiAVV3mR++hSdGqWHgkiaCjWaq78nJlgoNRa+6RRYD9SyNxP/89qJDirehIVxomlIFouChCMdoVkOqMckJZqPDcFEMnMrIgMsMdEmrZwJwV1+eZU0zsvuRdm9uyxUK2kcWTiGEyiCC1dQhVuoQR0IPMIzvMKb9WS9WO/Wx6I1Y6UzR/AH1ucPCnOR4Q==</latexit>
Sigmoid Hyperbolic Tangent (tanh)
Rectified Linear Unit (ReLU)
Initializing Network Weights
• Set all weights to 0?• Bad idea
• Set all weights to random values?• For very deep networks, gradients will vanish
• Main insight: want to keep variance of activations roughly same across layers• Xavier initialization – for tanh/sigmoid networks• He initialization – for ReLu networks• Both take into account fan-in/fan-out of each unit
Optimization Methods
Optimization Methods
• (Stochastic) gradient descent
• Stochastic gradient descent + momentum
• Adaptive gradient approaches:• RMSProp• Adam
w w � ↵rL(w)<latexit sha1_base64="v7tp3TuDdYwstovRhlZh+KIVUbU=">AAACJ3icbVBNS8NAEN34WetX1aOXxSLUgyVRwZ6k4MWDhwr2A5pSJttNu3SzCbsbSwn9N178K14EFdGj/8RNW7C2Phh4vDfDzDwv4kxp2/6ylpZXVtfWMxvZza3tnd3c3n5NhbEktEpCHsqGB4pyJmhVM81pI5IUAo/Tute/Tv36A5WKheJeDyPaCqArmM8IaCO1c1duALrn+clghF1OfQ1ShgM8o55iF3jUA+wK8Djg28KvedLO5e2iPQZeJM6U5NEUlXbu1e2EJA6o0ISDUk3HjnQrAakZ4XSUdWNFIyB96NKmoQICqlrJ+M8RPjZKB/uhNCU0HquzEwkESg0Dz3SmJ6p5LxX/85qx9kuthIko1lSQySI/5liHOA0Nd5ikRPOhIUAkM7di0gMJRJtosyYEZ/7lRVI7KzrnRefuIl8uTePIoEN0hArIQZeojG5QBVURQY/oGb2hd+vJerE+rM9J65I1nTlAf2B9/wB+RqY/</latexit>
z �z+rL(w)
w w � ↵z<latexit sha1_base64="KXz+/kDWIWOFwxXKDkKjXUnrqNE=">AAACWHicbVHBThsxEPUuUCClJcCRi9WoiKpqtAtIcETqhQMHkBpAiqNo1pklFl57Zc+Cwio/idRD+ytccEIQNPRJlp7fm7HHz1mplack+RPFC4tLH5ZXVhsf1z59Xm9ubF54WzmJHWm1dVcZeNTKYIcUabwqHUKRabzMbn5O/MtbdF5Z84tGJfYKuDYqVxIoSP2mFQXQMMvr+zHf4UJjTuCcveMiQwL+xv3OhYFMAz/dfVHvxt+EaLzu5k541X9wAbocAr/vN1tJO5mCvyfpjLTYDGf95oMYWFkVaEhq8L6bJiX1anCkpMZxQ1QeS5A3cI3dQA0U6Hv1NJgx/xqUAc+tC8sQn6pvO2oovB8VWaicDOvnvYn4P69bUX7Uq5UpK0Ijny/KK83J8knKfKAcStKjQEA6FWblcggOJIW/aIQQ0vknvycXe+10v52eH7SOj2ZxrLBt9oXtspQdsmN2ws5Yh0n2mz1Gi9FS9Ddm8XK8+lwaR7OeLfYP4s0nAoyzbA==</latexit>
Regularization – Classical Approaches
• Weight decay• Add an L2 term to cost function
• Early stopping• “Regularization in time”
Regularization – Newer Approaches
• Dropout
• Batch normalization• Motivation: “internal covariate shift”• Idea: Normalize activations at every layer
Summary
• Automatic differentiation• Better hardware + large datasets• Activation functions with better gradient flow• Heuristics for weight initialization• Better optimization algorithms• Batch normalization