GRID-LSTM

MAS.S63, Eric Chu

Neural Networks++Memory

Attention

Bayesian

Extremely deep

A LSTM RecapMotivation: problems with vanilla RNNs

Vanishing gradient due to non-linearities

Harder to capture longer term interactions

Solution:

“Long” “short-term” memory cells, controlled by gates that allow information to pass unmodified over many timesteps

A LSTM Recap

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

A LSTM Recap

Forget gate: which parts of memory vector to delete

Input gate: which parts of memory vector to update

Content gate: what should the memory vector be updated with

Output gate: what gets read from new memory into hidden vector

A LSTM Recap

A LSTM Recap

A LSTM Recap: Stacked LSTM

Grid-LSTM: MotivationStacked LSTM, but LSTM units connections along depth dimension as well as

temporal dimension

Grid-LSTM: Motivation

Grid-LSTM: 1D 1D Grid-LSTM = feedforward NN with LSTM cells instead of transfer functions

such as tanh and ReLU

Very closely related to Highway Networks

Grid-LSTM: 3D3D Grid-LSTM = Multidimensional LSTM, but again with LSTM cells in depth

dimension

2D Multidimensional RNN has 2 hidden vectors instead of 1

Grid-LSTM: All together now● N-D Grid-LSTM has N inputs and N outputs at each LSTM block

Relation to AttentionLSTM: “The mechanism also acts as a memory and implicit attention system,

whereby the signal from some input xi can be written to the memory vector and attended to in parts across multiple steps by being retrieved one part at a time.” - Quoc Le

Grid-LSTM: “Another interpretation of the attention model is that it allows an O(T) computation per prediction step. So the model itself has O(T2) total

computation (assuming the lengths of input and output sequences are roughly the same). With this interpretation, an alternative approach to the attention model is to lay out the input and output sequences in a grid structure to allow O(T2) computation. This idea is called Grid-LSTM” - Quoc Le

ExperimentTask: Character prediction

3-layer stacked LSTM vs. 3-layer stacked Grid-LSTM

Future WorkApplication to speech recognition, which

uses stacked RNNs on spectrograms

Start with 2D Grid-LSTM

Can also try 3D Grid-LSTM

Machine translation

3D Grid-LSTM instead of encoder decoder network