nip2015読み会「end-to-end memory networks」
TRANSCRIPT
-
NIPS2015 End-To-End Memory Networks
S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus
Preferred Infrastructure @unnonouno
2016/01/20 NIPS2015@
-
Memory networks
l 2013Facebook
l
lend-to-end
2
-
1
3
-
4
-
lChainer
l l 300
l Chainer
5
-
bAbI task
l
l 177
6
-
l l : {x1, x2, , xn} l : q l
l l : a
l l : A, B, C d x V l : W V x d l d: V:
7
-
8
-
l l F.sum(model.A(x), axis=1)
9
V
n
1 3 2 5 1x=
ID
=
-
l xiAmi
10
1 3 2 5 1x1=
4 3 1 7x2=
1 3 4 8 9x3=
m1m2m3m4
A
-
l Bu
11
B 3 4 1 7 9q =
u =
-
l miuSoftmaxpiAttention
l p = F.softmax(F.batch_matmul(m, u))
12
m1m2m3m4 u
p1p2p3p4 pi = softmax(miTu)
-
xiCci
13
1 3 2 5 1x1=
4 3 1 7x2=
1 3 4 8 9x3=
c1 c2 c3 c4
C
-
l cipio l o = F.batch_matmul(F.swapaxes(c ,2, 1), p)
14
p1p2p3p4
c1 c2 c3 c4x
=
o
-
l uoW
l loss = F.softmax_cross_entropy(model.W(u + o), a)
15
o u
+ W
-
lxiAmiCci lqBu l miusoftmaxpi
l cipio l o + uWa
softmax cross entropyloss
16
-
17
BoW
-
l
l l
18
-
19
-
l Adjacent l l Ak+t = Ck l B = A1
l piqx
l Layer-wise l A1 = A2 = l C1 = C2 =
20
-
temporal encoding
l
l
21
x1 = Sam walks into the kitchen x2 = Sam walks into the bedroom
q = Where is Sam?
-
20
22
3Adjacent
-
position encoding
l l
23
-
PE
24
-
lLinear start (LS) lsoftmax
l Random noise (RN) l10% l
25
-
26
-
l
l l
l
27
-
28
-
lend-to-end
l
l
29
-
lend-to-end
lattention
l6%
l
30