convolutional neural network & backpropagation … · 2017-11-09 · content 1.neural network...
TRANSCRIPT
![Page 1: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/1.jpg)
Convolutional Neural Network & Backpropagation Algorithm
Xuelin Qian
![Page 2: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/2.jpg)
Content
1.Neural Network
2.Backpropagation Algorithm
3.Convolutional Neural Network
4.Backpropagation on CNN
![Page 3: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/3.jpg)
Content
1.Neural Network
2.Backpropagation Algorithm
3.Convolutional Neural Network
4.Backpropagation on CNN
![Page 4: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/4.jpg)
“neuron”
where is called the activation function
1. Neural Network
![Page 5: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/5.jpg)
neural network: consist of many simple “neurons”
1. Neural Network
InputLayer
HiddenLayer
OutputLayer
![Page 6: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/6.jpg)
Forward Propagation
1. Neural Network
ln : the number of layerslL : the layer l( )lijW : the weight associated with the connection between unit in layer ,
and unit in layer j l
1+li( )lib : the bias associated with unit in layer 1+li( )lia : the acctivation unit in layeri l
![Page 7: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/7.jpg)
Forward Propagation
1. Neural Network
ln : the number of layerslL : the layer l( )lijW : the weight associated with the connection between unit in layer ,
and unit in layer j l
1+li( )lib : the bias associated with unit in layer 1+li( )lia : the acctivation unit in layeri
( )liz : the total weighted sum of inputs to unit in layer li
e.g.( ) ( ) ( )å
=
+=3
1
11
11
21
jjj bxWz( ) ( )( )212
1 zfa =
l
![Page 8: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/8.jpg)
Forward Propagation
1. Neural Network
ln : the number of layerslL : the layer l( )lijW : the weight associated with the connection between unit in layer ,
and unit in layer j l
1+li( )lib : the bias associated with unit in layer 1+li( )lia : the acctivation unit in layeri( )liz : the total weighted sum of inputs to unit in layer li
l
![Page 9: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/9.jpg)
Content
1.Neural Network
2.Backpropagation Algorithm
3.Convolutional Neural Network
4.Backpropagation on CNN
![Page 10: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/10.jpg)
2. Backpropagation Algorithm
“Loss function”
![Page 11: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/11.jpg)
2. Backpropagation Algorithm
“Gradient Descent”
![Page 12: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/12.jpg)
2. Backpropagation Algorithm
“Gradient Descent”
different i
![Page 13: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/13.jpg)
2. Backpropagation Algorithm
“Derivative chain rule”
( ) ( )( )11 ++ = li
li zfa (Page 7)
( )( ) ( )( ) ( )
( ) ( )( )( )
( )lij
liii
li
iilij W
zyxbWJz
yxbWJW ¶
¶´
¶¶
=¶¶ +
+
1
1 ,;,,;,
( ) ( ) ( ) ( )å=
+ +=ls
j
li
lj
lij
li baWz
1
1
![Page 14: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/14.jpg)
2. Backpropagation Algorithm
“error term”
( )( )
( ) ( )( )iili
li yxbWJ
z,;,1
1+
+
¶¶
=d
which measures how much that node was "responsible" for any errors in output
![Page 15: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/15.jpg)
2. Backpropagation Algorithm
“error term”
a) For each output unit in layer lni
( ) ( ) ( ) ( )å=
+ +=ls
j
li
lj
lij
li baWz
1
1( ) ( )( )11 ++ = li
li zfa (Page 7)
( )( )
( ) ( )( )iili
li yxbWJ
z,;,1
1+
+
¶¶
=d
![Page 16: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/16.jpg)
2. Backpropagation Algorithm
“error term”
b) For 2,,3,2,1 !---= lll nnnl
( ) ( ) ( ) ( )å=
+ +=ls
j
li
lj
lij
li baWz
1
1
( ) ( )( )11 ++ = li
li zfa
![Page 17: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/17.jpg)
2. Backpropagation Algorithm
“error term”
b) For 2,,3,2,1 !---= lll nnnl
![Page 18: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/18.jpg)
2. Backpropagation Algorithm
( )( ) ( )( ) ( )
( ) ( )( )( )
( )lij
liii
li
iilij W
zyxbWJz
yxbWJW ¶
¶´
¶¶
=¶¶ +
+
1
1 ,;,,;,
“Gradient Descent”
“Derivative chain rule”
“error term” ( )( )
( ) ( )( )iili
li yxbWJ
z,;,1
1+
+
¶¶
=d( ) ( ) ( ) ( )å
=
+ +=ls
j
li
lj
lij
li baWz
1
1
“Gradient Descent”
![Page 19: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/19.jpg)
2. Backpropagation Algorithm
“error term”
![Page 20: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/20.jpg)
2. Backpropagation Algorithm
“error term”
![Page 21: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/21.jpg)
2. Backpropagation Algorithm
“error term”
结论:一个点(神经元)的残差=前向的时候下一层中使用过该点的神经单元的残差按权重求和
![Page 22: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/22.jpg)
Content
1.Neural Network
2.Backpropagation Algorithm
3.Convolutional Neural Network
4.Backpropagation on CNN
![Page 23: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/23.jpg)
3. Convolutional Neural Network
![Page 24: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/24.jpg)
3. Convolutional Neural Network
![Page 25: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/25.jpg)
3. Convolutional Neural Network
convolution
How to calculate the number of hidden units?1,000,000 hidden units?
![Page 26: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/26.jpg)
3. Convolutional Neural Network
convolution layer
stride, pad, kernel(size), number(output)
1_
)_ker_2(1 +
-´+=+ hstride
hnelhpadHH ll 1
_)_ker_2(
1 +-´+
=+ wstridewnelwpadWW l
l
3_ker_ker == wnelhnel
5== ll WH
0=pad
1=stride
![Page 27: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/27.jpg)
3. Convolutional Neural Network
pooling layer
Max Pooling & Avg Pooling
![Page 28: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/28.jpg)
3. Convolutional Neural Network
fully connected layer
activation function
ReLU Sigmoid tanh
![Page 29: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/29.jpg)
3. Convolutional Neural Network
MNIST
![Page 30: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/30.jpg)
3. Convolutional Neural Network
convolution layer
a. compute the input size
b. setup stride, pad, kernel(size), number(output),
c. compute the size of filter and output
d. *initialize weights (the first iteration)
e. convolution
al,
![Page 31: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/31.jpg)
Content
1.Neural Network
2.Backpropagation Algorithm
3.Convolutional Neural Network
4.Backpropagation on CNN
![Page 32: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/32.jpg)
4. Backpropagation on CNN
Ø Backpropagation on pooling layer
Ø Backpropagation on convolution layer
![Page 33: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/33.jpg)
4. Backpropagation on CNN
“error term”
结论:一个点(神经元)的残差=前向的时候下一层中使用过该点的神经单元的残差按权重求和
![Page 34: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/34.jpg)
4. Backpropagation on CNN
Backpropagation on pooling layer
1 2 3 45 6 7 89 2 3 56 5 1 1
6 8
9 5
max pooling
kernel_size=2stride=2pad=0
Forward
0 0 0 00 1 0 23 0 0 40 0 0 0
1 2
3 4Forward
( ) ( )ll dd ®+1
![Page 35: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/35.jpg)
4. Backpropagation on CNN
Backpropagation on pooling layer
1 2 3 4
5 6 7 8
9 2 3 5
6 5 1 1
7 11
11 5
avg pooling
kernel_size=2stride=2pad=0
Forward
0.25 0.25 0.5 0.5
0.25 0.25 0.5 0.5
0.75 0.75 1 1
0.75 0.75 1 1
1 2
3 4Backward ( ) ( )ll dd ®+1
![Page 36: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/36.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
1 1 1 0 0
0 0 0 1 0
0 1 1 1 1
0 1 1 0 0
1 1 1 1 0
w1 w2 w3
w4 w5 w6
w7 w8 w9
cstride=1pad=0
lW 1+ll
![Page 37: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/37.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=?
lW ( )padl ++1d ld
![Page 38: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/38.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=2
lW ld
结论:一个点(神经元)的残差=前向的时候下一层中使用过该点的神经单元的残差按权重求和
c
( )padl ++1d
![Page 39: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/39.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=?
lW ld( )padl ++1d
c
![Page 40: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/40.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=?
lW ld
c
( )padl ++1d
![Page 41: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/41.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=?
lW ld
c
( )padl ++1d
![Page 42: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/42.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=?
lW ld
c
( )padl ++1d
![Page 43: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/43.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=2
lW ld( )padl ++1d
![Page 44: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/44.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
w1 w2 w3
w4 w5 w6
w7 w8 w9
c
0 0 0 0 0 0 00 0 0 0 0 0 00 0 1 2 3 0 00 0 4 5 6 0 00 0 7 8 9 0 00 0 0 0 0 0 00 0 0 0 0 0 0
1)ker2(1 +
-´+=+ stride
nelpadWW ll
stride=1pad=2
lW ld( )padl ++1d
(rotation)
![Page 45: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/45.jpg)
4. Backpropagation on CNN
Backpropagation on convolution layer
a.
b. rotate
c. compute (convolution)
d. compute gradient
e. update weights
padl ++1dlW °180
ld
JW¶¶
![Page 46: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/46.jpg)
Reference
1. UFLDL: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial2. Blogs:
http://www.cnblogs.com/tornadomeet/tag/Deep%20Learning/http://blog.csdn.net/zouxy09/article/category/1387932
![Page 47: Convolutional Neural Network & Backpropagation … · 2017-11-09 · Content 1.Neural Network 2.Backpropagation Algorithm 3.Convolutional Neural Network 4.Backpropagation on CNN](https://reader031.vdocuments.mx/reader031/viewer/2022022600/5b3de6b77f8b9a560a8e5614/html5/thumbnails/47.jpg)
THANK YOU