bbm495-lecture8burcucan/bbm495-lecture8-4pp.pdf · exp(uwvc) dot product compares similarity of o...

4/29/19

1

BBM 495WORD EMBEDDINGS

2018-2019 SPRING

§ How similar is pizza to pasta?§ How related is pizza to Italy?

§ Representing words as vectors allows easy computation of similarity

4/29/19

2

§ Increase in size with vocabulary§ Very high dimensional: require a lot of storage§ Subsequent classification models have sparsity issues§ Models are less robust

4/29/19

3

4/29/19

4

4/29/19

5

4/29/19

6

4/29/19

7

Predict!

4/29/19

8

§ Remember: two vectors are similar if they have a high dot product§ Cosine is just a normalized dot product

§ So: § Similarity (o,c) ∝ uo · Vc

§ Wewill need to normalize to get a probability

§ We use softmax to turn into probabilities:

4/29/19

9

4/29/19

10

4/29/19

11

all§ Take gradients at each window

§ Go through gradient for each center vector v in a window§ In each window, we will compute updates for all parameters that

are being used in that window. For example:

4/29/19

12

4/29/19

13

4/29/19

14

4/29/19

15

§ Dense Vectors, Dan Jurafsky§ Representation for Language: from Word Embeddings to

Sentence Meanings, Christopher Manning, Stanford University, 2017

§ Natural Language Processing with Deep Learning, Richard Socher, Stanford University

§ More Word Vectors, Richard Socher, Stanford University§ Improving Distributional Similarity with Lessons Learned from

Word Embeddings, Omer Levy,