Recurrent and Recursive Networks

Marco Kuhlmann

Neural Networks with Applications to Vision and Language

Introduction

Applications of sequence modelling

• Map unsegmented connected handwriting to strings.

• Map sequences of acoustic signals to sequences of phonemes.

• Translate sentences from one language into another one.

• Generate baby names, poems, source code, patent applications.

The bag-of-words model

The gorgeously elaborate continuation of “The Lord of the Rings” trilogy is so huge that a column of words cannot adequately describe co-writer/director Peter Jackson’s expanded vision of J.R.R. Tolkien’s Middle-earth.

… is a sour little movie at its core; an exploration of the emptiness that underlay the relentless gaiety of the 1920’s, as if to stop would hasten the economic and global political turmoil that was to come.

pos neg

The bag-of-words model

a adequately cannot co-writer column continuation describe director elaborate expanded gorgeously huge is J.R.R. Jackson Lord Middle-earth of of of of Peter Rings so that The The the Tolkien trilogy vision words

1920’s a an and as at come core economic emptiness exploration gaiety global hasten if is its little movie of of political relentless sour stop that that the the the the to to turmoil underlay was would

pos neg

Part-of-speech tagging

jag bad om en kort bit

PN VB PP DT JJ NN

NN NN

PL RG

AB VBSN PN

NN

AB NN

Hidden Markov Models (HMMs)

jag bad om en kort bit

PN VB PP DT JJ NN

𝑃(PN|BOS)𝑃(VB|PN)

𝑃(PP|VB)𝑃(DT|PP)

𝑃(JJ|DT)𝑃(NN|JJ)

𝑃(EOS|NN)

𝑃(jag|PN) 𝑃(om|PP) 𝑃(bit|NN)𝑃(en|DT)𝑃(bad|VB) 𝑃(kort|JJ)

transition probabilities, emission probabilities

VB

PN

BOS EOS

𝑃(VB| BOS) 𝑃(EOS|VB)

𝑃(PN| BOS) 𝑃(EOS|PN)

𝑃(PN|VB) 𝑃(VB|PN)

𝑃(VB|VB)

𝑃(PN|PN)

𝑃(𝑤 |VB)

𝑃(𝑤 |PN)

A weakness of Hidden Markov Models

• The only information that an HMM has access to at any given point in time is its state.

• Suppose that the HMM has 𝑛 states. Then the number of the current state can be written using log 𝑛 bits.

• Thus the current state contains at most log 𝑛 bits of information about the sequence generated so far.

Strengths of recurrent neural networks

• Distributed hidden state

In recurrent neural networks, several units can be active at once, which allows them to store a lot of information efficiently. contrast that with one current state in HMMs

• Non-linear dynamics

Different units can interact with each other in non-linear ways. This makes recurrent neural networks Turing-complete. contrast that with linear dynamical systems

Attribution: Geoffrey Hinton

Recurrent neural networks (RNNs)

• Recurrent neural networks can be visualised as networks with feedback connections, which form directed cycles between units.

• These feedback connections are unfolded over time.

• A crucial property of recurrent neural networks is that they share the same set of parameters across different timesteps.

RNN, cyclic representation

𝒐

𝒙

= delay of one timestep𝒇

𝒉

RNN, unrolled

𝒐(2)

𝒙(2)

𝒐(1) 𝒐(3)

𝒙(1) 𝒙(3)

𝒉(2) 𝒉(3)𝒉(1)𝒇 𝒇𝒇 𝒇

General observations

• The parameters of the model are shared across all timesteps.

• The hidden state can be influenced by the entire input seen so far. Contrast this with the Markov assumption of HMMs.

• The hidden state can be a ‘lossy summary’ of the input sequence. Hopefully, this state will encode useful information for the task at hand.

• The model has the same input size regardless of sequence length. specified in terms of transitions from one state to the other

Different types of RNN architectures

encoder

generator

transducer

Training recurrent neural networks

Computation graph for a standard architecture

𝒚(2)𝒚(1) 𝒚(3)

𝑳(2)𝑳(1) 𝑳(3)

𝒙(2)𝒙(1) 𝒙(3)

𝑾 𝑾𝑾 𝑾𝒉(2) 𝒉(3)𝒉(1)

𝑽

𝑼 𝑼 𝑼

𝑽 𝑽

𝒐(1) 𝒐(2) 𝒐(3)

Assumptions

• The hidden states are computed by some nonlinear activation function, such as tanh.

• The outputs at each time step are normalised log-probabilities representing distributions over a finite set of labels. In the book, this is assumed to happen implicitly when computing the loss.

Backpropagation through time

• Unrolled recurrent neural networks are just feedforward networks, and can therefore be trained using backpropagation. parameter sharing; linear constraints on the parameters

• This way of training recurrent neural networks is called backpropagation through time.

• Given that the unrolled computation graphs can be very deep, the vanishing gradient problem is exacerbated in RNNs.

𝑦𝑖

𝐸

𝑤𝑖𝑗

𝑤𝑗𝑘

𝑓

𝑓

𝑦𝑘𝑧𝑘

𝑦𝑗𝑧𝑗

𝑡

𝑦𝑖

𝐸

𝑤𝑖𝑗

𝑤𝑗𝑘

𝑓

𝑓

𝑦𝑘𝑧𝑘

𝑦𝑗𝑧𝑗

𝑡

Backpropagation through time

𝒚(2)𝒚(1) 𝒚(3)

𝑳(2)𝑳(1) 𝑳(3)

𝒙(2)𝒙(1) 𝒙(3)

𝑾 𝑾𝑾 𝑾𝒉(2) 𝒉(3)𝒉(1)

𝑽

𝑼 𝑼 𝑼

𝑽 𝑽

𝒐(1) 𝒐(2) 𝒐(3)

Initial values of the hidden state

• We could manually specify the initial state in terms of some sensible starting value.

• We could learn the initial state by starting with a random guess and then updating that guess during backpropagation.

Networks with output recurrence

𝒚(2)𝒚(1) 𝒚(3)

𝑳(2)𝑳(1) 𝑳(3)

𝑾 𝑾𝑾 𝑾

𝒉(2) 𝒉(3)𝒉(1)

𝑽 𝑽 𝑽

𝒙(2)𝒙(1) 𝒙(3)

𝑼 𝑼 𝑼

𝒐(1) 𝒐(2) 𝒐(3)

The limitations of recurrent neural networks

• In principle, recurrent networks are capable of learning long-distance dependencies.

• In practice, standard gradient-based learning algorithms do not perform very well. Bengio et al. (1994) – the ‘vanishing gradient’ problem

• Today, there are several methods available for training recurrent neural networks that avoids these problems. LSTMs, optimisation with small gradients, careful weight initialisations, …

Vanishing and exploding gradients

-1

-0,5

0

0,5

1

-6 -3 0 3 6

sigmoid tanh relu

0

0,25

0,5

0,75

1

-6 -3 0 3 6

sigmoid tanh relu

activation functions gradients

Recursive neural networks

𝒙(2)𝒙(1) 𝒙(3) 𝒙(4)

𝒐 𝒚

𝑳

𝑽 𝑽 𝑽 𝑽

𝑼 𝑾 𝑼 𝑾

𝑼 𝑾

Long Short-Term Memory (LSTM)

Long Short-Term Memory

• The Long Short-Term Memory (LSTM) architecture was specifically designed to battle the vanishing gradients problem

• Metaphor: The dynamic state of the neural network can be considered as a short-term memory.

• The LSTM architecture tries to make this short-term memory last as long as possible by preventing vanishing gradients.

• Central idea: gating mechanism

Memory cell and gating mechanism

The crucial innovation in an LSTM is the design of its memory cell.

• Information is written into the cell whenever its ‘write’ gate is on.

• The information stays in the cell as long as its ‘keep’ gate is on.

• Information is read from the cell whenever its ‘read’ gate is on.

Information flow in an LSTM

write write write

keep keep keep keep

write write write

1.7

1.7 1.7 1.7

1.7

Attribution: Geoffrey Hinton

time →

A look inside an LSTM cell

𝒙(𝑖)

𝒚(𝑖)

𝒉(𝑖−1)

𝒔(𝑖−1)

𝒉(𝑖)

𝒔(𝑖)

𝜎 tanh𝜎 𝜎

tanh

• +

••

Attr

ibut

ion:

Chr

is O

lah

The ‘keep’ gate (‘forget gate’)

𝒙(𝑖)

𝜎

𝒉(𝑖−1)

Attr

ibut

ion:

Chr

is O

lah

The ‘write’ gate (‘input gate’)

𝜎

𝒙(𝑖)

𝒉(𝑖−1)

Attr

ibut

ion:

Chr

is O

lah

Update candidate

𝒙(𝑖)

𝒉(𝑖−1)tanh

Attr

ibut

ion:

Chr

is O

lah

Updating the internal state

𝒔(𝑖−1) 𝒔(𝑖)• +

𝒙(𝑖)

•

Attr

ibut

ion:

Chr

is O

lah

The ‘write’ gate (‘output gate’)

𝒙(𝑖)

𝒉(𝑖−1)𝜎

Attr

ibut

ion:

Chr

is O

lah

Updating the external state

𝒚(𝑖)

𝒉(𝑖)

tanh

•

Attr

ibut

ion:

Chr

is O

lah

Peephole connections

𝒙𝑖

𝒚𝑖

𝒔𝑖−1 𝒔𝑖

𝜎 tanh𝜎 𝜎

tanh

× +

××

Attr

ibut

ion:

Chr

is O

lah

Gated Recurrent Unit (GRU)

𝒙(𝑖)

𝒉(𝑖−1) 𝒉(𝑖)

tanh𝜎

1−•

• +

•

𝜎

Attr

ibut

ion:

Chr

is O

lah

Bidirectional RNNs

• In speech recognition, the correct interpretation of a given sound may depend on both the previous sounds and the next sounds.

• Bidirectional RNNs combine one RNN that moves forward through time with another RNN that moves backward.

• The output can be a representation that depends on both the past and the future, without having to specify a fixed-sized window.

A bidirectional RNN

𝒚(2)

𝒙(2)

𝒚(1) 𝒚(3)

𝒙(1) 𝒙(3)

F B

𝒉(1) 𝒉(2)

𝒈(2) 𝒈(3)

Attr

ibut

ion:

Chr

is O

lah