improving distributional similarity with lessons learned from word embeddings omer levy yoav...

Improving Distributional Similaritywith Lessons Learned from Word

Embeddings

Omer Levy Yoav Goldberg Ido Dagan

Bar-Ilan UniversityIsrael

Word Similarity & Relatedness

• How similar is pizza to pasta?• How related is pizza to Italy?

• Representing words as vectors allows easy computation of similarity

Approaches for Representing Words

Distributional Semantics (Count)• Used since the 90’s• Sparse word-context PMI/PPMI matrix• Decomposed with SVD

Word Embeddings (Predict)• Inspired by deep learning• word2vec (Mikolov et al., 2013)• GloVe (Pennington et al., 2014)

Underlying Theory: The Distributional Hypothesis (Harris, ’54; Firth, ‘57)“Similar words occur in similar contexts”

Approaches for Representing Words

Both approaches:• Rely on the same linguistic theory• Use the same data• Are mathematically related• “Neural Word Embedding as Implicit Matrix Factorization” (NIPS 2014)

• How come word embeddings are so much better?• “Don’t Count, Predict!” (Baroni et al., ACL 2014)

• More than meets the eye…

What’s really improving performance?

The Contributions of Word EmbeddingsNovel Algorithms(objective + training method)

• Skip Grams + Negative Sampling• CBOW + Hierarchical Softmax• Noise Contrastive Estimation• GloVe• …

New Hyperparameters(preprocessing, smoothing, etc.)

• Subsampling• Dynamic Context Windows• Context Distribution Smoothing• Adding Context Vectors• …

Our Contributions

1) Identifying the existence of new hyperparameters• Not always mentioned in papers

2) Adapting the hyperparameters across algorithms• Must understand the mathematical relation between algorithms

Our Contributions

1) Identifying the existence of new hyperparameters• Not always mentioned in papers

2) Adapting the hyperparameters across algorithms• Must understand the mathematical relation between algorithms

3) Comparing algorithms across all hyperparameter settings• Over 5,000 experiments

Background

What is word2vec?

How is it related to PMI?

What is word2vec?

• word2vec is not a single algorithm• It is a software package for representing words as vectors, containing:• Two distinct models

• CBoW• Skip-Gram

• Various training methods• Negative Sampling• Hierarchical Softmax

• A rich preprocessing pipeline• Dynamic Context Windows• Subsampling• Deleting Rare Words

What is word2vec?

• word2vec is not a single algorithm• It is a software package for representing words as vectors, containing:• Two distinct models

• CBoW• Skip-Gram (SG)

• Various training methods• Negative Sampling (NS)• Hierarchical Softmax

• A rich preprocessing pipeline• Dynamic Context Windows• Subsampling• Deleting Rare Words

Skip-Grams with Negative Sampling (SGNS)Marco saw a furry little wampimuk hiding in the tree.

“word2vec Explained…”Goldberg & Levy, arXiv 2014

words contextswampimuk furrywampimuk littlewampimuk hidingwampimuk in… …

(data)

Skip-Grams with Negative Sampling (SGNS)• SGNS finds a vector for each word in our vocabulary • Each such vector has latent dimensions (e.g. )• Effectively, it learns a matrix whose rows represent • Key point: it also learns a similar auxiliary matrix of context vectors• In fact, each word has two embeddings

𝑊𝑑

𝑉𝑊

:wampimuk

𝐶𝑉𝐶

:wampimuk

Skip-Grams with Negative Sampling (SGNS)

Skip-Grams with Negative Sampling (SGNS)• Maximize: • was observed with

words contextswampimuk furrywampimuk littlewampimuk hidingwampimuk in

Skip-Grams with Negative Sampling (SGNS)• Maximize: • was observed with

words contextswampimuk furrywampimuk littlewampimuk hidingwampimuk in

• Minimize: • was hallucinated with

words contextswampimuk Australiawampimuk cyberwampimuk thewampimuk 1985

Skip-Grams with Negative Sampling (SGNS)• “Negative Sampling”• SGNS samples contexts at random as negative examples• “Random” = unigram distribution

• Spoiler: Changing this distribution has a significant effect

What is SGNS learning?

• Take SGNS’s embedding matrices ( and )

“Neural Word Embeddings as Implicit Matrix Factorization”Levy & Goldberg, NIPS 2014

𝑊𝑑

𝑉𝑊

𝑉𝐶

• Take SGNS’s embedding matrices ( and )• Multiply them• What do you get?

𝑊𝑑

𝑉𝑊

𝐶𝑉 𝐶

• A matrix• Each cell describes the relation between a specific word-context pair

𝑊𝑑

𝑉𝑊

𝐶𝑉 𝐶

𝑉𝑊

𝑉 𝐶

• We proved that for large enough and enough iterations

𝑊𝑑

𝑉𝑊

𝐶𝑉 𝐶

𝑉𝑊

𝑉 𝐶

• We proved that for large enough and enough iterations• We get the word-context PMI matrix

𝑊𝑑

𝑉𝑊

𝐶𝑉 𝐶

𝑀𝑃𝑀𝐼¿

𝑉𝑊

𝑉 𝐶

• We prove that for large enough and enough iterations• We get the word-context PMI matrix, shifted by a global constant

𝑊𝑑

𝑉𝑊

𝐶𝑉 𝐶

𝑀𝑃𝑀𝐼¿

𝑉𝑊

𝑉 𝐶

− log𝑘

• SGNS is doing something very similar to the older approaches

• SGNS is factorizing the traditional word-context PMI matrix

• So does SVD!

• GloVe factorizes a similar word-context matrix

But embeddings are still better, right?• Plenty of evidence that embeddings outperform traditional methods• “Don’t Count, Predict!” (Baroni et al., ACL 2014)• GloVe (Pennington et al., EMNLP 2014)

• How does this fit with our story?

The Big Impact of “Small” Hyperparameters

The Big Impact of “Small” Hyperparameters• word2vec & GloVe are more than just algorithms…

• Introduce new hyperparameters

• May seem minor, but make a big difference in practice

Identifying New Hyperparameters

New Hyperparameters

• Preprocessing (word2vec)• Dynamic Context Windows• Subsampling• Deleting Rare Words

• Postprocessing (GloVe)• Adding Context Vectors

• Association Metric (SGNS)• Shifted PMI• Context Distribution Smoothing

New Hyperparameters

Dynamic Context Windows

Marco saw a furry little wampimuk hiding in the tree.

word2vec:

GloVe:

Aggressive:

The Word-Space Model (Sahlgren, 2006)

Adding Context Vectors

• SGNS creates word vectors • SGNS creates auxiliary context vectors • So do GloVe and SVD

Adding Context Vectors

• SGNS creates word vectors • SGNS creates auxiliary context vectors • So do GloVe and SVD

• Instead of just • Represent a word as:

• Introduced by Pennington et al. (2014)• Only applied to GloVe

Adapting Hyperparameters across Algorithms

Context Distribution Smoothing

• SGNS samples to form negative examples

• Our analysis assumes is the unigram distribution

• SGNS samples to form negative examples

• Our analysis assumes is the unigram distribution

• In practice, it’s a smoothed unigram distribution

• This little change makes a big difference

• We can adapt context distribution smoothing to PMI!

• Replace with :

• Consistently improves PMI on every task

• Always use Context Distribution Smoothing!

Comparing Algorithms

Controlled Experiments

• Prior art was unaware of these hyperparameters

• Essentially, comparing “apples to oranges”

• We allow every algorithm to use every hyperparameter

Controlled Experiments

• Prior art was unaware of these hyperparameters

• Essentially, comparing “apples to oranges”

• We allow every algorithm to use every hyperparameter*

* If transferable

Systematic Experiments

• 9 Hyperparameters• 6 New

• 4 Word Representation Algorithms• PPMI (Sparse & Explicit)• SVD(PPMI)• SGNS• GloVe

• 8 Benchmarks• 6 Word Similarity Tasks• 2 Analogy Tasks

• 5,632 experiments

Systematic Experiments

• 9 Hyperparameters• 6 New

• 4 Word Representation Algorithms• PPMI (Sparse & Explicit)• SVD(PPMI)• SGNS• GloVe

• 8 Benchmarks• 6 Word Similarity Tasks• 2 Analogy Tasks

• 5,632 experiments

Hyperparameter Settings

Classic Vanilla Setting(commonly used for distributional baselines)

• Preprocessing• <None>

• Postprocessing• <None>

• Association Metric• Vanilla PMI/PPMI

Hyperparameter Settings

Classic Vanilla Setting(commonly used for distributional baselines)

• Preprocessing• <None>

• Association Metric• Vanilla PMI/PPMI

Recommended word2vec Setting(tuned for SGNS)

• Preprocessing• Dynamic Context Window• Subsampling

• Association Metric• Shifted PMI/PPMI• Context Distribution Smoothing

Experiments

PPMI (Sparse Vectors) SGNS (Embeddings)0.3

WordSim-353 Relatedness

’s Co

Experiments: Prior Art

VanillaSetting

word2vecSetting

’s Co

Experiments: “Apples to Apples”Experiments: “Oranges to Oranges”

Experiments: “Oranges to Oranges”Experiments: Hyperparameter Tuning

VanillaSetting

word2vecSetting

OptimalSetting

’s Co

[different settings]

Overall Results

• Hyperparameters often have stronger effects than algorithms

• Hyperparameters often have stronger effects than more data

• Prior superiority claims were not accurate

Re-evaluating Prior Claims

Don’t Count, Predict! (Baroni et al., 2014)• “word2vec is better than count-based methods”

• Hyperparameter settings account for most of the reported gaps

• Embeddings do not really outperform count-based methods

Don’t Count, Predict! (Baroni et al., 2014)• “word2vec is better than count-based methods”

• Hyperparameter settings account for most of the reported gaps

• Embeddings do not really outperform count-based methods*

* Except for one task…

GloVe (Pennington et al., 2014)

• “GloVe is better than word2vec”

• Hyperparameter settings account for most of the reported gaps• Adding context vectors applied only to GloVe• Different preprocessing

• We observed the opposite• SGNS outperformed GloVe on every task

• Our largest corpus: 10 billion tokens• Perhaps larger corpora behave differently?

GloVe (Pennington et al., 2014)

• “GloVe is better than word2vec”

• Hyperparameter settings account for most of the reported gaps• Adding context vectors applied only to GloVe• Different preprocessing

• We observed the opposite• SGNS outperformed GloVe on every task

• Our largest corpus: 10 billion tokens• Perhaps larger corpora behave differently?

Linguistic Regularities in Sparse and ExplicitWord Representations (Levy and Goldberg, 2014)

• “PPMI vectors perform on par with SGNS on analogy tasks”

• Holds for semantic analogies• Does not hold for syntactic analogies (MSR dataset)

• Hyperparameter settings account for most of the reported gaps• Different context type for PPMI vectors

• Syntactic Analogies: there is a real gap in favor of SGNS

Conclusions

Conclusions: Distributional Similarity

The Contributions of Word Embeddings:• Novel Algorithms• New Hyperparameters

What’s really improving performance?• Hyperparameters (mostly)• The algorithms are an improvement• SGNS is robust

Conclusions: Distributional Similarity

The Contributions of Word Embeddings:• Novel Algorithms• New Hyperparameters

What’s really improving performance?• Hyperparameters (mostly)• The algorithms are an improvement• SGNS is robust & efficient

Conclusions: Methodology

• Look for hyperparameters

• Adapt hyperparameters across different algorithms

• For good results: tune hyperparameters

• For good science: tune baselines’ hyperparameters

Thank you :)

improving distributional similarity with lessons learned from word embeddings omer levy yoav...

Documents

dagan 12" ember retainer for fireplace grates

word2vec embeddings: cbow and skipgram · word2vec...

google analytics basics - dagan varagic

dagan wells

crm and ecommerce. yoav kutner

this talk - stanford...

magen yoav

dr. yoav mazeh

book a table yoav 2014 - pdf

dagan dagan agricultural automation ltd. e-mail:...

dagan wells (1)

1.dagan 001.50

ido dagan bar- ilan university, israel

sagi dagan

yoav lerman thesis

dense word embeddings

3.dagan 101.150

yoav liberman's portable bench

improving distributional similarity with lessons...

catalogo dagan 2012