Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3996–4007July 5 - 10, 2020. c©2020 Association for Computational Linguistics

3996

BERTRAM: Improved Word Embeddings Have Big Impact onContextualized Model Performance

Timo SchickSulzer GmbH

Munich, Germanytimo.schick@sulzer.de

Hinrich SchutzeCenter for Information and Language Processing

LMU Munich, Germanyinquiries@cislmu.org

Abstract

Pretraining deep language models has led tolarge performance gains in NLP. Despite thissuccess, Schick and Schutze (2020) recentlyshowed that these models struggle to under-stand rare words. For static word embeddings,this problem has been addressed by separatelylearning representations for rare words. Inthis work, we transfer this idea to pretrainedlanguage models: We introduce BERTRAM, apowerful architecture based on BERT that iscapable of inferring high-quality embeddingsfor rare words that are suitable as input rep-resentations for deep language models. This isachieved by enabling the surface form and con-texts of a word to interact with each other in adeep architecture. Integrating BERTRAM intoBERT leads to large performance increasesdue to improved representations of rare andmedium frequency words on both a rare wordprobing task and three downstream tasks.1

1 Introduction

As word embedding algorithms (e.g. Mikolov et al.,2013) are known to struggle with rare words, sev-eral techniques for improving their representationshave been proposed. These approaches exploit ei-ther the contexts in which rare words occur (Lazari-dou et al., 2017; Herbelot and Baroni, 2017; Kho-dak et al., 2018; Liu et al., 2019a), their surface-form (Luong et al., 2013; Bojanowski et al., 2017;Pinter et al., 2017), or both (Schick and Schutze,2019a,b; Hautte et al., 2019). However, all of thisprior work is designed for and evaluated on uncon-textualized word embeddings.

Contextualized representations obtained frompretrained deep language models (e.g. Peters et al.,2018; Radford et al., 2018; Devlin et al., 2019; Liuet al., 2019b) already handle rare words implicitly

1Our implementation of BERTRAM is publicly available athttps://github.com/timoschick/bertram.

using methods such as byte-pair encoding (Sen-nrich et al., 2016), WordPiece embeddings (Wuet al., 2016) and character-level CNNs (Baevskiet al., 2019). Nevertheless, Schick and Schutze(2020) recently showed that BERT’s (Devlin et al.,2019) performance on a rare word probing task canbe significantly improved by explicitly learning rep-resentations of rare words using Attentive Mimick-ing (AM) (Schick and Schutze, 2019a). However,AM is limited in two important respects:

• For processing contexts, it uses a simple bag-of-words model, making poor use of the avail-able information.

• It combines form and context in a shallowfashion, preventing both input signals frominteracting in a complex manner.

These limitations apply not only to AM, but to allprevious work on obtaining representations for rarewords by leveraging form and context. While usingbag-of-words models is a reasonable choice forstatic embeddings, which are often themselves bag-of-words (e.g. Mikolov et al., 2013; Bojanowskiet al., 2017), it stands to reason that they are notthe best choice to generate input representationsfor position-aware, deep language models.

To overcome these limitations, we introduceBERTRAM (BERT for Attentive Mimicking), anovel architecture for learning rare word representa-tions that combines a pretrained BERT model withAM. As shown in Figure 1, the learned rare wordrepresentations can then be used as an improvedinput representation for another BERT model. Bygiving BERTRAM access to both surface form andcontexts starting at the lowest layer, a deep integra-tion of both input signals becomes possible.

Assessing the effectiveness of methods likeBERTRAM in a contextualized setting is challeng-ing: While most previous work on rare words was

3997

evaluated on datasets explicitly focusing on rarewords (e.g Luong et al., 2013; Herbelot and Ba-roni, 2017; Khodak et al., 2018; Liu et al., 2019a),these datasets are tailored to uncontextualized em-beddings and thus not suitable for evaluating ourmodel. Furthermore, rare words are not well repre-sented in commonly used downstream task datasets.We therefore introduce rarification, a procedure toautomatically convert evaluation datasets into onesfor which rare words are guaranteed to be impor-tant. This is achieved by replacing task-relevantfrequent words with rare synonyms obtained usingsemantic resources such as WordNet (Miller, 1995).We rarify three common text (or text pair) classifica-tion datasets: MNLI (Williams et al., 2018), AG’sNews (Zhang et al., 2015) and DBPedia (Lehmannet al., 2015). BERTRAM outperforms previouswork on four English datasets by a large margin:on the three rarified datasets and on WNLaMPro(Schick and Schutze, 2020).

In summary, our contributions are as follows:

• We introduce BERTRAM, a model that inte-grates BERT into Attentive Mimicking, en-abling a deep integration of surface-form andcontexts and much better representations forrare words.

• We devise rarification, a method that trans-forms evaluation datasets into ones for whichrare words are guaranteed to be important.

• We show that adding BERTRAM to BERTachieves a new state-of-the-art on WNLaM-Pro (Schick and Schutze, 2020) and beats allbaselines on rarified AG’s News, MNLI andDBPedia, resulting in an absolute improve-ment of up to 25% over BERT.

2 Related Work

Surface-form information (e.g., morphemes, char-acters or character n-grams) is commonly used toimprove word representations. For static word em-beddings, this information can either be injectedinto a given embedding space (Luong et al., 2013;Pinter et al., 2017), or a model can directly be givenaccess to it during training (Bojanowski et al., 2017;Salle and Villavicencio, 2018; Piktus et al., 2019).In the area of contextualized representations, manyarchitectures employ subword segmentation meth-ods (e.g. Radford et al., 2018; Devlin et al., 2019;Yang et al., 2019; Liu et al., 2019b). Others use

riding a un ##ic ##y ##cle is hard

BERT

ariding is hard

BERT

BERTRAMBERTRAM

unicycle

Figure 1: Top: Standard use of BERT. Bottom: Ourproposal; first BERTRAM learns an embedding for “uni-cycle” that replaces the WordPiece sequence. BERT isthen run on this improved input representation.

convolutional neural networks to directly accesscharacter-level information (Kim et al., 2016; Pe-ters et al., 2018; Baevski et al., 2019).

Complementary to surface form, another usefulsource of information for understanding rare wordsare the contexts in which they occur (Lazaridouet al., 2017; Herbelot and Baroni, 2017; Khodaket al., 2018). Schick and Schutze (2019a,b) showthat combining form and context leads to signifi-cantly better results than using just one of the two.While all of these methods are bag-of-words mod-els, Liu et al. (2019a) recently proposed an architec-ture based on context2vec (Melamud et al., 2016).However, in contrast to our work, they (i) do notincorporate surface-form information and (ii) donot directly access the hidden states of context2vec,but instead simply use its output distribution.

Several datasets focus on rare words, e.g., Stan-ford Rare Word (Luong et al., 2013), DefinitionalNonce (Herbelot and Baroni, 2017), and Contex-tual Rare Word (Khodak et al., 2018). However,unlike our rarified datasets, they are only suitablefor evaluating uncontextualized word representa-tions. Rarification is related to adversarial examplegeneration (e.g. Ebrahimi et al., 2018), which ma-nipulates the input to change a model’s prediction.We use a similar mechanism to determine whichwords in a given sentence are most important andreplace them with rare synonyms.

3 Model

3.1 Form-Context ModelWe first review the basis for our new model, theform-context model (FCM) (Schick and Schutze,2019b). Given a set of d-dimensional high-qualityembeddings for frequent words, FCM induces em-beddings for rare words that are appropriate for

3998

the given embedding space. This is done as fol-lows: Given a word w and a context C in whichit occurs, a surface-form embedding vform

(w,C) ∈ Rd

is obtained by averaging over embeddings of allcharacter n-grams in w; the n-gram embeddingsare learned during training. Similarly, a contextembedding vcontext

(w,C) ∈ Rd is obtained by averagingover the embeddings of all words in C. Finally,both embeddings are combined using a gate

g(vform(w,C), v

context(w,C) ) = σ(x>[vform

(w,C); vcontext(w,C) ] + y)

with parameters x ∈ R2d, y ∈ R and σ denotingthe sigmoid function, allowing the model to decidehow to weight surface-form and context. The finalrepresentation of w is then a weighted combinationof form and context embeddings:

v(w,C) = α · (Avcontext(w,C) + b) + (1− α) · vform

(w,C)

where α = g(vform(w,C), v

context(w,C) ) and A ∈ Rd×d, b ∈

Rd are parameters learned during training.The context part of FCM is able to capture the

broad topic of rare words, but since it is a bag-of-words model, it is not capable of obtaining a moreconcrete or detailed understanding (see Schick andSchutze, 2019b). Furthermore, the simple gatingmechanism results in only a shallow combinationof form and context. That is, the model is notable to combine form and context until the verylast step: While it can learn to weight form andcontext components, the two embeddings (formand context) do not share any information and thusdo not influence each other.

3.2 BERTRAM

To overcome these limitations, we introduceBERTRAM, a model that combines a pretrainedBERT language model (Devlin et al., 2019) withAttentive Mimicking (Schick and Schutze, 2019a).We denote with et the (uncontextualized, i.e., first-layer) embedding assigned to a (wordpiece) tokent by BERT. Given a sequence of such uncontextu-alized embeddings e = e1, . . . , en, we denote byhj(e) the contextualized representation of the j-thtoken at the final layer when the model is given eas input.

Given a word w and a context C in which it oc-curs, let t = t1, . . . , tm be the sequence obtainedfrom C by (i) replacing w with a [MASK] tokenand (ii) tokenization (matching BERT’s vocabu-lary); furthermore, let i denote the index for which

ti = [MASK]. We experiment with three variantsof BERTRAM: BERTRAM-SHALLOW, BERTRAM-REPLACE and BERTRAM-ADD.2

SHALLOW. Perhaps the simplest approach forobtaining a context embedding fromC using BERTis to define

vcontext(w,C) = hi(et1 , . . . , etm) .

This approach aligns well with BERT’s pretrain-ing objective of predicting likely substitutes for[MASK] tokens from their contexts. The contextembedding vcontext

(w,C) is then combined with its formcounterpart as in FCM.

While this achieves our first goal of using a moresophisticated context model that goes beyond bag-of-words, it still only combines form and contextin a shallow fashion.

REPLACE. Before computing the context embed-ding, we replace the uncontextualized embeddingof the [MASK] token with the word’s surface-formembedding:

vcontext(w,C) = hi(et1 , ... , eti−1 , v

form(w,C), eti+1 , ... , etm) .

Our rationale for this is as follows: During regularBERT pretraining, words chosen for prediction arereplaced with [MASK] tokens only 80% of thetime and kept unchanged 10% of the time. Thus,standard pretrained BERT should be able to makeuse of form embeddings presented this way as theyprovide a strong signal with regards to how the“correct” embedding of w may look like.

ADD. Before computing the context embedding,we prepad the input with the surface-form embed-ding of w, followed by a colon (e:):3

vcontext(w,C) = hi+2(v

form(w,C), e:, et1 , . . . , etm) .

The intuition behind this third variant is that lex-ical definitions and explanations of a word w areoccasionally prefixed by “w :” (e.g., in some on-line dictionaries). We assume that BERT has seenmany definitional sentences of this kind during pre-training and is thus able to leverage surface-forminformation about w presented this way.

For both REPLACE and ADD, surface-form in-formation is directly and deeply integrated into the

2We refer to these three BERTRAM configurations simplyas SHALLOW, REPLACE and ADD.

3We experimented with other prefixes, but found that thisvariant is best capable of recovering w at the masked position.

3999

〈S〉was wash . . . les〈S〉

vform(w,C1)

e[CLS] e: eother e[MASK] esuch eas etrousers . . .

: other [MASK] such as trousers . . .

BERT

A · + b

v(w,C1)

BERTRAM . . . BERTRAM

v(w,C1) . . . v(w,Cm)

(w,C1) . . . (w,Cm)

Attentive Mimicking

v(w,C)

Figure 2: Schematic representation of BERTRAM-ADD processing the input word w = “washables” given a singlecontext C1 = “other washables such as trousers . . .” (left) and given multiple contexts C = {C1, . . . , Cm} (right)

computation of the context embedding; thus, wedo not require any gating mechanism and directlyset v(w,C) = A · vcontext

(w,C) + b. Figure 2 (left) showshow a single context is processed using ADD.

To exploit multiple contexts of a word if avail-able, we follow the approach of Schick and Schutze(2019a) and add an AM layer on top of our model;see Figure 2 (right). Given a set of contextsC = {C1, . . . , Cm} and the corresponding em-beddings v(w,C1), . . . , v(w,Cm), AM applies a self-attention mechanism to all embeddings, allowingthe model to distinguish informative from uninfor-mative contexts. The final embedding v(w,C) is thena weighted combination of all embeddings:

v(w,C) =∑m

i=1ρi · v(w,Ci)

where the self-attention layer determines theweights ρi subject to

∑mi=1 ρi = 1. For further

details, see Schick and Schutze (2019a).

3.3 Training

Like previous work, we use mimicking (Pinter et al.,2017) as a training objective. That is, given a fre-quent word w with known embedding ew and a setof corresponding contexts C, BERTRAM is trainedto minimize ‖ew − v(w,C)‖2.

Training BERTRAM end-to-end is costly: thecost of processing a single training instance (w, C)with C = {C1, . . . , Cm} is the same as processingan entire batch of m examples in standard BERT.Therefore, we resort to the following three-stagetraining process:

1. We train only the context part, minimizing‖ew − A · (

∑mi=1 ρi · vcontext

(w,Ci)) + b‖2 where

ρi is the weight assigned to each context Ci

through the AM layer. Regardless of the se-lected BERTRAM variant, the context embed-ding is always obtained using SHALLOW inthis stage. Furthermore, only A, b and allparameters of the AM layer are optimized.

2. We train only the form part (i.e., only the n-gram embeddings); our loss for a single exam-ple (w, C) is ‖ew − vform

(w,C)‖2. Training in this

stage is completely detached from the under-lying BERT model.

3. In the third stage, we combine the pretrainedform-only and context-only models and trainall parameters. The first two stages are onlyrun once and then used for all three BERTRAM

variants because context and form are trainedin isolation. The third stage must be run foreach variant separately.

We freeze all of BERT’s parameters during trainingas we – somewhat surprisingly – found that thisslightly improves the model’s performance whilespeeding up training. For ADD, we additionallyfound it helpful to freeze the form part in the thirdtraining stage. Importantly, for the first two stagesof our training procedure, we do not have to back-propagate through BERT to obtain all required gra-dients, drastically increasing the training speed.

4 Dataset Rarification

The ideal dataset for measuring the quality of rareword representations would be one for which theaccuracy of a model with no understanding of rarewords is 0% whereas the accuracy of a model thatperfectly understands rare words is 100%. Unfortu-nately, existing datasets do not satisfy this desidera-

4000

tum, not least because rare words – by their nature– occur rarely.

This does not mean that rare words are not im-portant: As we shift our focus in NLP from wordsand sentences as the main unit of processing tolarger units like paragraphs and documents, rarewords will occur in a high proportion of such larger“evaluation units”. Rare words are also clearly ahallmark of human language competence, whichshould be the ultimate goal of NLP. Our work ispart of a trend that sees a need for evaluation tasksin NLP that are more ambitious than what we havenow.4

To create more challenging datasets, we use rar-ification, a procedure that automatically transformsexisting text classification datasets in such a waythat rare words become important. We require apretrained language model M as a baseline, anarbitrary text classification dataset D containing la-beled instances (x, y) and a substitution dictionaryS, mapping each word w to a set of rare synonymsS(w). Given these ingredients, our procedure con-sists of three steps: (i) splitting the dataset into atrain set and a set of test candidates, (ii) training thebaseline model on the train set and (iii) modifyinga subset of the test candidates to generate the finaltest set.

Dataset Splitting. We partition D into a trainingset Dtrain and a set of test candidates, Dcand. Dcandcontains all instances (x, y) ∈ D such that for atleast one word w in x, S(w) 6= ∅ – subject to theconstraint that the training set contains at least onethird of the entire data.

Baseline Training. We finetuneM onDtrain. Let(x, y) ∈ Dtrain where x = w1, . . . , wn is a se-quence of words. We deviate from the finetuningprocedure of Devlin et al. (2019) in three respects:

• We randomly replace 5% of all words in xwith a [MASK] token. This allows the modelto cope with missing or unknown words, aprerequisite for our final test set generation.

• As an alternative to overwriting the languagemodel’s uncontextualized embeddings for rarewords, we also want to allow models to add analternative representation during test time, in

4Cf. (Bowman, 2019): “If we want to be able to establishfair benchmarks that encourage future progress toward robust,human-like language understanding, we’ll need to get betterat creating clean, challenging, and realistic test datasets.”

which case we simply separate both represen-tations by a slash (cf. §5.3). To accustom thelanguage model to this duplication of words,we replace each word wi with “wi / wi” witha probability of 10%. To make sure that themodel does not simply learn to always focuson the first instance during training, we ran-domly mask each of the two repetitions withprobability 25%.

• We do not finetune the model’s embeddinglayer. We found that this does not hurt per-formance, an observation in line with recentfindings of Lee et al. (2019).

Test Set Generation. Let p(y | x) be the proba-bility that the finetuned model M assigns to class ygiven input x, and M(x) = argmaxy∈Y p(y | x)be the model’s prediction for input x where Y de-notes the set of all labels. For generating our testset, we only consider candidates that are classifiedcorrectly by the baseline model, i.e., candidates(x, y) ∈ Dcand with M(x) = y. For each suchentry, let x = w1, . . . , wn and let xwi=t be the se-quence obtained from x by replacing wi with t. Wecompute

wi = argminwj :S(wj)6=∅

p(y | xwj=[MASK]),

i.e., we select the word wi whose masking pushesthe model’s prediction the farthest away fromthe correct label. If removing this word al-ready changes the model’s prediction – that is,M(xwi=[MASK]) 6= y –, we select a random raresynonym wi ∈ S(wi) and add (xwi=wi

, y) to thetest set. Otherwise, we repeat the above procedure;if the label still has not changed after masking up to5 words, we discard the candidate. Each instance(xwi1

=wi1,...,wik

=wik, y) of the resulting test set has

the following properties:

• If each wij is replaced by [MASK], the entryis classified incorrectly by M . In other words,understanding the words wij is necessary forM to determine the correct label.

• If the model’s internal representation of eachwij is sufficiently similar to its representationof wij , the entry is classified correctly by M .That is, if the model is able to understandthe rare words wij and to identify them assynonyms of wij , it will predict the correctlabel.

4001

Model RARE MEDIUM

BERT (base) 0.112 0.234+ AM (Schick and Schutze, 2020) 0.251 0.267+ BERTRAM-SHALLOW 0.250 0.246+ BERTRAM-REPLACE 0.155 0.216+ BERTRAM-ADD 0.269 0.367BERT (large) 0.143 0.264

RoBERTa (large) 0.270 0.275+ BERTRAM-ADD 0.306 0.323

Table 1: MRR on WNLaMPro test for baseline mod-els and various BERTRAM configurations. Best resultsper base model are underlined, results that do not dif-fer significantly from the best results in a paired t-test(p < 0.05) are bold.

Note that the test set is closely coupled to thebaseline model M because we select the words tobe replaced based on M ’s predictions. Importantly,however, the model is never queried with any raresynonym during test set generation, so its repre-sentations of rare words are not taken into accountfor creating the test set. Thus, while the test setis not suitable for comparing M with an entirelydifferent modelM ′, it allows us to compare variousstrategies for representing rare words in the embed-ding space ofM . Definitional Nonce (Herbelot andBaroni, 2017) is subject to a similar constraint: itis tied to a specific (uncontextualized) embeddingspace based on Word2Vec (Mikolov et al., 2013).

5 Evaluation

5.1 Setup

For our evaluation of BERTRAM, we follow the ex-perimental setup of Schick and Schutze (2020). Weexperiment with integrating BERTRAM both intoBERTbase and RoBERTalarge (Liu et al., 2019b).Throughout our experiments, when BERTRAM isused to provide input representations for one of thetwo models, we use the same model as BERTRAM’sunderlying language model. Further training speci-fications can be found in Appendix A.

While BERT was trained on BookCorpus (Zhuet al., 2015) and a large Wikipedia dump, we fol-low previous work and train BERTRAM only on themuch smaller Westbury Wikipedia Corpus (WWC)(Shaoul and Westbury, 2010); this of course givesBERT a clear advantage over BERTRAM. This ad-vantage is even more pronounced when comparingBERTRAM with RoBERTa, which is trained on acorpus that is an order of magnitude larger than theoriginal BERT corpus. We try to at least partially

Task Entry

MNLI i think i will go finish up my laundry wash-ables.

AG’s [. . . ] stake will improve meliorate syman-tec’s consulting contacts [. . . ]

DBPedia yukijiro hotaru [. . . ] is a japanese nipponeseactor histrion.

MNLI a smart person is often ofttimes correct intheir answers ansers.

MNLI the southwest has a lot of farming andvineyards vineries that make excellentfantabulous merlot.

Table 2: Examples from rarified datasets. Crossed out:replaced words. Bold: replacements.

compensate for this as follows: In our downstreamtask experiments, we gather the set of contexts Cfor each word from WWC+BookCorpus duringinference.5

5.2 WNLaMProWe evaluate BERTRAM on the WNLaMPro dataset(Schick and Schutze, 2020). This dataset consistsof cloze-style phrases like “A lingonberry is a .”and the task is to correctly fill the slot ( ) withone of several acceptable target words (e.g., “fruit”,“bush” or “berry”), which requires understanding ofthe meaning of the phrase’s keyword (“lingonberry”in the example). As the goal of this dataset is toprobe a language model’s ability to understand rarewords without any task-specific finetuning, Schickand Schutze (2020) do not provide a training set.The dataset is partitioned into three subsets basedon the keyword’s frequency in WWC: RARE (oc-curring fewer than 10 times) MEDIUM (occurringbetween 10 and 100 times), and FREQUENT (allremaining words).

For our evaluation, we compare the performanceof a standalone BERT (or RoBERTa) model withone that uses BERTRAM as shown in Figure 1 (bot-tom). As our focus is to improve representationsfor rare words, we evaluate our model only on WN-LaMPro RARE and MEDIUM. Table 1 gives results;our measure is mean reciprocal rank (MRR). Wesee that supplementing BERT with any of the pro-posed methods results in noticeable improvementsfor the RARE subset, with ADD clearly outperform-ing SHALLOW and REPLACE. Moreover, ADD per-forms surprisingly well for more frequent words,improving the score for WNLaMPro-MEDIUM by

5We recreate BookCorpus with the script at github.com/soskek/bookcorpus. We refer to the joined cor-pus of WWC and BookCorpus as WWC+BookCorpus.

4002

MNLI AG’s News DBPedia

Model All Msp WN All Msp WN All Msp WN

BERT (base) 50.5 49.1 53.4 56.5 54.8 61.9 49.3 46.0 57.6+ Mimick (Pinter et al., 2017) 37.2 38.2 38.7 45.3 43.9 50.5 36.5 35.8 41.1+ A La Carte (Khodak et al., 2018) 44.6 45.7 46.1 52.4 53.7 56.1 51.1 48.7 59.3+ AM (Schick and Schutze, 2020) 50.9 50.7 53.6 58.9 59.8 62.6 60.7 63.1 62.8+ BERTRAM 53.3 52.5 55.6 62.1 63.1 65.3 64.2 67.9 64.1+ BERTRAM-SLASH 56.4 55.3 58.6 62.9 63.3 65.3 65.7 67.3 67.2+ BERTRAM-SLASH + INDOMAIN 59.8 57.3 62.7 62.5 62.1 66.6 74.2 74.8 76.7

RoBERTa (large) 67.3 68.7 68.4 63.7 68.1 65.7 65.5 67.3 66.6+ BERTRAM-SLASH 70.1 71.5 70.9 64.6 68.4 64.9 71.9 73.8 73.9+ BERTRAM-SLASH + INDOMAIN 71.7 71.9 73.2 68.1 71.9 69.0 76.0 78.8 77.3

Table 3: Accuracy of standalone BERT and RoBERTa, various baselines and BERTRAM on rarified MNLI, AG’sNews and DBPedia. The five BERTRAM instances are BERTRAM-ADD. Best results per baseline model areunderlined, results that do not differ significantly from the best results in a two-sided binomial test (p < 0.05) arebold. Msp/WN: subset of instances containing at least one misspelling/synonym. All: all instances.

58% compared to BERTbase and 37% comparedto Attentive Mimicking. This makes sense con-sidering that the key enhancement of BERTRAM

over AM lies in improving context representationsand interconnection of form and context; the morecontexts are given, the more this comes into play.Noticeably, despite being both based on and in-tegrated into a BERTbase model, our architectureeven outperforms BERTlarge by a large margin.While RoBERTa performs much better than BERTon WNLaMPro, BERTRAM still significantly im-proves results for both rare and medium frequencywords. As it performs best for both the RARE andMEDIUM subset, we always use the ADD configura-tion of BERTRAM in the following experiments.

5.3 Downstream Task DatasetsTo measure the effect of adding BERTRAM to apretrained deep language model on downstreamtasks, we rarify (cf. §4) the following three datasets:

• MNLI (Williams et al., 2018), a natural lan-guage inference dataset where given two sen-tences a and b, the task is to decide whethera entails b, a and b contradict each other orneither;

• AG’s News (Zhang et al., 2015), a news classi-fication dataset with four different categories(world, sports, business and science/tech);

• DBPedia (Lehmann et al., 2015), an ontologydataset with 14 classes (e.g., company, artist)that have to be identified from text snippets.

For all three datasets, we create rarified instancesboth using BERTbase and RoBERTalarge as a base-line model and build the substitution dictionary S

using the synonym relation of WordNet (Miller,1995) and the pattern library (Smedt and Daele-mans, 2012) to make sure that all synonyms haveconsistent parts of speech. Furthermore, we onlyconsider synonyms for each word’s most frequentsense; this filters out much noise and improves thequality of the created sentences. In addition toWordNet, we use the misspelling dataset of Pik-tus et al. (2019). To prevent misspellings fromdominating the resulting datasets, we only assignmisspelling-based substitutes to randomly selected10% of the words contained in each sentence. Mo-tivated by the results on WNLaMPro-MEDIUM, weconsider every word that occurs less than 100 timesin WWC+BookCorpus as being rare. Exampleentries from the rarified datasets obtained usingBERTbase as a baseline model can be seen in Ta-ble 2. The average number of words replaced withsynonyms or misspellings is 1.38, 1.82 and 2.34for MNLI, AG’s News and DBPedia, respectively.

Our default way of injecting BERTRAM embed-dings into the baseline model is to replace the se-quence of uncontextualized subword token embed-dings for a given rare word with its BERTRAM-based embedding (Figure 1, bottom). That is,given a sequence of uncontextualized token em-beddings e = e1, . . . , en where ei, . . . , ej with1 ≤ i ≤ j ≤ n is the sequence of embeddingsfor a single rare word w with BERTRAM-basedembedding v(w,C), we replace e with

e′ = e1, . . . , ei−1, v(w,C), ej+1, . . . , en .

As an alternative to replacing the original se-quence of subword embeddings for a given rareword, we also consider BERTRAM-SLASH, a con-

4003

figuration where the BERTRAM-based embeddingis simply added and both representations are sepa-rated using a single slash:

eSLASH = e1, . . . , ej , e/, v(w,C), ej+1, . . . , en .

The intuition behind this variant is that in BERT’spretraining corpus, a slash is often used to separatetwo variants of the same word (e.g., “useable / us-able”) or two closely related concepts (e.g., “com-pany / organization”, “web-based / cloud”) andthus, BERT should be able to understand that bothei, . . . , ej and v(w,C) refer to the same entity. Wetherefore surmise that whenever some informationis encoded in one representation but not in the other,giving BERT both representations is helpful.

By default, the set of contexts C for eachword is obtained by collecting all sentences fromWWC+BookCorpus in which it occurs. We alsotry a variant where we add in-domain contexts bygiving BERTRAM access to all texts (but not la-bels) found in the test set; we refer to this variant asINDOMAIN.6 Our motivation for including this vari-ant is as follows: Moving from the training stage ofa model to its production use often causes a slightdomain shift. This is turn leads to an increasednumber of input sentences containing words thatdid not – or only very rarely – appear in the trainingdata. However, such input sentences can easily becollected as additional unlabeled examples duringproduction use. While there is no straightforwardway to leverage these unlabeled examples with analready finetuned BERT model, BERTRAM can eas-ily make use of them without requiring any labelsor any further training: They can simply be in-cluded as additional contexts during inference. Asthis gives BERTRAM a slight advantage, we alsoreport results for all configurations without usingindomain data. Importantly, adding indomain dataincreases the number of contexts for more than 90%of all rare words by at most 3, meaning that theycan still be considered rare despite the additionalindomain contexts.

Table 3 reports, for each task, the accuracy on theentire dataset (All) as well as scores obtained con-sidering only instances where at least one word wasreplaced by a misspelling (Msp) or a WordNet syn-onym (WN), respectively.7 Consistent with results

6For the MNLI dataset, which consists of text pairs (a, b),we treat a and b as separate contexts.

7Note that results for BERT and RoBERTa are only looselycomparable because the datasets generated from both baselinemodels through rarification are different.

1 2 4 8 16 32 64 128

40

45

50

55

60

65

BERT

BERT+BSLBERT

BERT+BSL

BERT

BERT+BSL

cmax

Acc

urac

y

MNLIAG’s NewsDBPedia

Figure 3: BERT vs. BERT combined with BERTRAM-SLASH (BERT+BSL) on three downstream tasks forvarying maximum numbers of contexts cmax

on WNLaMPro, combining BERT with BERTRAM

consistently outperforms both a standalone BERTmodel and one combined with various baselinemodels. Using the SLASH variant brings improve-ments across all datasets as does adding INDOMAIN

contexts (exception: BERT/AG’s News). Thismakes sense considering that for a rare word, everysingle additional context can be crucial for gaininga deeper understanding. Correspondingly, it is notsurprising that the benefit of adding BERTRAM toRoBERTa is less pronounced, because BERTRAM

uses only a fraction of the contexts available toRoBERTa during pretraining. Nonetheless, addingBERTRAM significantly improves RoBERTa’s ac-curacy for all three datasets both with and withoutadding INDOMAIN contexts.

To further understand for which words usingBERTRAM is helpful, Figure 3 looks at the accuracyof BERTbase both with and without BERTRAM as afunction of word frequency. That is, we computethe accuracy scores for both models when consid-ering only entries (xwi1

=wi1,...,wik

=wik, y) where

each substituted word wij occurs less than cmaxtimes in WWC+BookCorpus, for different valuesof cmax. As one would expect, cmax is positively cor-related with the accuracies of both models, showingthat the rarer a word is, the harder it is to under-stand. Interestingly, the gap between standaloneBERT and BERT with BERTRAM remains moreor less constant regardless of cmax. This suggeststhat using BERTRAM may even be helpful for morefrequent words.

To investigate this hypothesis, we perform an-other rarification of MNLI that differs from the

4004

[0,125) [125,250) [250,500) [500,∞)

0

2

4

6

8

10

Word counts

Acc

urac

yim

prov

emen

t BERT+BSL RoBERTa+BSL

BERT+BSL+ID RoBERTa+BSL+ID

Figure 4: Improvements for BERT (base) andRoBERTa (large) when adding BERTRAM-SLASH(+BSL) or BERTRAM-SLASH + INDOMAIN (+BSL+ID)on MNLI-1000

previous rarification in two respects. First, we in-crease the threshold for a word to count as rarefrom 100 to 1000. Second, as this means that wehave more WordNet synonyms available, we do notuse the misspelling dictionary (Piktus et al., 2019)for substitution. We refer to the resulting datasetsfor BERTbase and RoBERTalarge as MNLI-1000.

Figure 4 shows results on MNLI-1000 for var-ious rare word frequency ranges. For each value[c0, c1) on the x-axis, the y-axis shows improve-ment in accuracy compared to standalone BERTor RoBERTa when only dataset entries are con-sidered for which each rarified word occurs be-tween c0 (inclusively) and c1 (exclusively) timesin WWC+BooksCorpus. We see that for wordswith frequency less than 125, the improvement inaccuracy remains similar even without using mis-spellings as another source of substitutions. In-terestingly, for every single interval of rare wordcounts considered, adding BERTRAM-SLASH toBERT considerably improves its accuracy. ForRoBERTa, adding BERTRAM brings improvementsonly for words occurring less than 500 times.While using INDOMAIN data is beneficial forrare words – simply because it gives us addi-tional contexts for these words –, when consid-ering only words that occur at least 250 times inWWC+BookCorpus, adding INDOMAIN contextsdoes not help.

6 Conclusion

We have introduced BERTRAM, a novel architec-ture for inducing high-quality representations for

rare words in BERT’s and RoBERTa’s embeddingspaces. This is achieved by employing a powerfulpretrained language model and deeply integratingsurface-form and context information. By replac-ing important words with rare synonyms, we cre-ated downstream task datasets that are more chal-lenging and support the evaluation of NLP modelson the task of understanding rare words, a capa-bility that human speakers have. On all of thesedatasets, BERTRAM improves over standard BERTand RoBERTa, demonstrating the usefulness of ourmethod.

Our analysis showed that BERTRAM is benefi-cial not only for rare words (our main target in thispaper), but also for frequent words. In future work,we want to investigate BERTRAM’s potential bene-fits for such frequent words. Furthermore, it wouldbe interesting to explore more complex ways ofincorporating surface-form information – e.g., byusing a character-level CNN similar to the one ofKim et al. (2016) – to balance out the potency ofBERTRAM’s form and context parts.

Acknowledgments

This work was funded by the European ResearchCouncil (ERC #740516). We would like to thankthe anonymous reviewers for their helpful com-ments.

ReferencesAlexei Baevski, Sergey Edunov, Yinhan Liu, Luke

Zettlemoyer, and Michael Auli. 2019. Cloze-drivenpretraining of self-attention networks. In Proceed-ings of the 2019 Conference on Empirical Methodsin Natural Language Processing and the 9th Inter-national Joint Conference on Natural Language Pro-cessing (EMNLP-IJCNLP), pages 5359–5368, HongKong, China. Association for Computational Lin-guistics.

Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. 2017. Enriching word vectors withsubword information. Transactions of the Associa-tion for Computational Linguistics, 5:135–146.

Sam Bowman. 2019. Google T5 explores the limits oftransfer learning.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),

4005

pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and DejingDou. 2018. HotFlip: White-box adversarial exam-ples for text classification. In Proceedings of the56th Annual Meeting of the Association for Compu-tational Linguistics (Volume 2: Short Papers), pages31–36, Melbourne, Australia. Association for Com-putational Linguistics.

Jeroen Van Hautte, Guy Emerson, and Marek Rei.2019. Bad form: Comparing context-based andform-based few-shot learning in distributional se-mantic models. Computing Research Repository,arXiv:1910.00275.

Aurelie Herbelot and Marco Baroni. 2017. High-risklearning: acquiring new word vectors from tiny data.In Proceedings of the 2017 Conference on Empiri-cal Methods in Natural Language Processing, pages304–309. Association for Computational Linguis-tics.

Mikhail Khodak, Nikunj Saunshi, Yingyu Liang,Tengyu Ma, Brandon Stewart, and Sanjeev Arora.2018. A la carte embedding: Cheap but effectiveinduction of semantic feature vectors. In Proceed-ings of the 56th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers), pages 12–22. Association for ComputationalLinguistics.

Yoon Kim, Yacine Jernite, David Sontag, and Alexan-der M. Rush. 2016. Character-aware neural lan-guage models. In Proceedings of the Thirtieth AAAIConference on Artificial Intelligence, AAAI’16,pages 2741–2749. AAAI Press.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In 3rd Inter-national Conference on Learning Representations(ICLR).

Angeliki Lazaridou, Marco Marelli, and Marco Baroni.2017. Multimodal word meaning induction fromminimal exposure to natural text. Cognitive Science,41(S4):677–705.

Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019.What would Elsa do? Freezing layers during trans-former fine-tuning. Computing Research Reposi-tory, arXiv:1911.03090.

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch,Dimitris Kontokostas, Pablo N. Mendes, SebastianHellmann, Mohamed Morsey, Patrick van Kleef,Soren Auer, and Christian Bizer. 2015. DBpedia -a large-scale, multilingual knowledge base extractedfrom wikipedia. Semantic Web Journal, 6(2):167–195.

Qianchu Liu, Diana McCarthy, and Anna Korhonen.2019a. Second-order contexts from lexical substi-tutes for few-shot learning of word representations.In Proceedings of the Eighth Joint Conference on

Lexical and Computational Semantics (*SEM 2019),pages 61–67, Minneapolis, Minnesota. Associationfor Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019b.RoBERTa: A robustly optimized BERT pretrain-ing approach. Computing Research Repository,arXiv:1907.11692.

Thang Luong, Richard Socher, and Christopher Man-ning. 2013. Better word representations with re-cursive neural networks for morphology. In Pro-ceedings of the Seventeenth Conference on Computa-tional Natural Language Learning, pages 104–113,Sofia, Bulgaria. Association for Computational Lin-guistics.

Oren Melamud, Jacob Goldberger, and Ido Dagan.2016. context2vec: Learning generic context em-bedding with bidirectional LSTM. In Proceedingsof The 20th SIGNLL Conference on ComputationalNatural Language Learning, pages 51–61, Berlin,Germany. Association for Computational Linguis-tics.

Tomas Mikolov, Kai Chen, Greg Corrado, and JeffreyDean. 2013. Efficient estimation of word represen-tations in vector space. Computing Research Repos-itory, arXiv:1301.3781.

George A. Miller. 1995. Wordnet: A lexical databasefor english. Commun. ACM, 38(11):39–41.

Adam Paszke, Sam Gross, Soumith Chintala, GregoryChanan, Edward Yang, Zachary DeVito, ZemingLin, Alban Desmaison, Luca Antiga, and AdamLerer. 2017. Automatic differentiation in PyTorch.In NIPS Autodiff Workshop.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. In Proceedings of the 2018 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long Papers), pages2227–2237, New Orleans, Louisiana. Associationfor Computational Linguistics.

Aleksandra Piktus, Necati Bora Edizel, Piotr Bo-janowski, Edouard Grave, Rui Ferreira, and FabrizioSilvestri. 2019. Misspelling oblivious word embed-dings. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages3226–3234, Minneapolis, Minnesota. Associationfor Computational Linguistics.

Yuval Pinter, Robert Guthrie, and Jacob Eisenstein.2017. Mimicking word embeddings using subwordRNNs. In Proceedings of the 2017 Conference onEmpirical Methods in Natural Language Processing,

4006

pages 102–112. Association for Computational Lin-guistics.

Alec Radford, Karthik Narasimhan, Tim Salimans, andIlya Sutskever. 2018. Improving language under-standing by generative pre-training.

Alexandre Salle and Aline Villavicencio. 2018. Incor-porating subword information into matrix factoriza-tion word embeddings. In Proceedings of the Sec-ond Workshop on Subword/Character LEvel Models,pages 66–71, New Orleans. Association for Compu-tational Linguistics.

Timo Schick and Hinrich Schutze. 2019a. Attentivemimicking: Better word embeddings by attendingto informative contexts. In Proceedings of the 2019Conference of the North American Chapter of theAssociation for Computational Linguistics: HumanLanguage Technologies, Volume 1 (Long and ShortPapers), pages 489–494, Minneapolis, Minnesota.Association for Computational Linguistics.

Timo Schick and Hinrich Schutze. 2019b. Learning se-mantic representations for novel words: Leveragingboth form and context. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence.

Timo Schick and Hinrich Schutze. 2020. Rare words:A major problem for contextualized embeddings andhow to fix it by attentive mimicking. In Proceedingsof the Thirty-Fourth AAAI Conference on ArtificialIntelligence.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Neural machine translation of rare wordswith subword units. In Proceedings of the 54th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computa-tional Linguistics.

Cyrus Shaoul and Chris Westbury. 2010. The westburylab wikipedia corpus.

Tom De Smedt and Walter Daelemans. 2012. Patternfor python. Journal of Machine Learning Research,13(Jun):2063–2067.

Adina Williams, Nikita Nangia, and Samuel Bowman.2018. A broad-coverage challenge corpus for sen-tence understanding through inference. In Proceed-ings of the 2018 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume1 (Long Papers), pages 1112–1122. Association forComputational Linguistics.

Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Transformers: State-of-the-art natural language processing.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V.Le, Mohammad Norouzi, Wolfgang Macherey,Maxim Krikun, Yuan Cao, Qin Gao, KlausMacherey, Jeff Klingner, Apurva Shah, Melvin John-son, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws,Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, KeithStevens, George Kurian, Nishant Patil, Wei Wang,Cliff Young, Jason Smith, Jason Riesa, Alex Rud-nick, Oriol Vinyals, Greg Corrado, Macduff Hughes,and Jeffrey Dean. 2016. Google’s neural machinetranslation system: Bridging the gap between hu-man and machine translation. Computing ResearchRepository, arXiv:1609.08144.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car-bonell, Ruslan Salakhutdinov, and Quoc V Le.2019. XLNet: Generalized autoregressive pretrain-ing for language understanding. Computing Re-search Repository, arXiv:1906.08237.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.Character-level convolutional networks for text clas-sification. In C. Cortes, N. D. Lawrence, D. D. Lee,M. Sugiyama, and R. Garnett, editors, Advances inNeural Information Processing Systems 28, pages649–657. Curran Associates, Inc.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut-dinov, Raquel Urtasun, Antonio Torralba, and SanjaFidler. 2015. Aligning books and movies: Towardsstory-like visual explanations by watching moviesand reading books. In Proceedings of the IEEE inter-national conference on computer vision, pages 19–27.

4007

A Training Details

Our implementation of BERTRAM is based on Py-Torch (Paszke et al., 2017) and the Transform-ers library (Wolf et al., 2019). To obtain tar-get embeddings for frequent multi-token words(i.e., words that occur at least 100 times inWWC+BookCorpus) during training, we use one-token approximation (OTA) (Schick and Schutze,2020). For RoBERTalarge, we found increasing thenumber of iterations per word from 4,000 to 8,000to produce better OTA embeddings using the sameevaluation setup as Schick and Schutze (2020). Forall stages of training, we use Adam (Kingma andBa, 2015) as optimizer.

Context-Only Training. During the first stageof our training process, we train BERTRAM with amaximum sequence length of 96 and a batch sizeof 48 contexts for BERTbase and 24 contexts forRoBERTalarge. These parameters are chosen suchthat a batch fits on a single Nvidia GeForce GTX1080Ti. Each context in a batch is mapped to aword w from the set of training words, and eachbatch contains at least 4 and at most 32 contexts perword. For BERTbase and RoBERTalarge, we pretrainthe context part for 5 and 3 epochs, respectively.We use a maximum learning rate of 5 · 10−5 andperform linear warmup for the first 10% of trainingexamples, after which the learning rate is linearlydecayed.

Form-Only Training. In the second stage of ourtraining process, we use the same parameters asSchick and Schutze (2020), as our form-only modelis the very same as theirs. That is, we use a learningrate of 0.01, a batch size of 64 words and we applyn-gram dropout with a probability of 10%. Wepretrain the form-only part for 20 epochs.

Combined Training. For the final stage, we usethe same training configuration as for context-onlytraining, but we keep n-gram dropout from theform-only stage. We perform combined training for3 epochs. For ADD, when using RoBERTa as an un-derlying language model, we do not just prepad theinput with the surface-form embedding followedby a colon, but additionally wrap the surface-formembedding in double quotes. That is, we prepadthe input with e”, vform

(w,C), e”, e:. We found this toperform slightly better in preliminary experimentswith some toy examples.

B Evaluation Details

WNLaMPro In order to ensure comparabilitywith results of Schick and Schutze (2020), we useonly WWC to obtain contexts for WNLaMPro key-words.

Rarified Datasets To obtain rarified instancesof MNLI, AG’s News and DBPedia, we trainBERTbase and RoBERTalarge on each task’s train-ing set for 3 epochs. We use a batch size of 32,a maximum sequence length of 128 and a weightdecay factor of 0.01. For BERT, we perform linearwarmup for the first 10% of training examples anduse a maximum learning rate of 5 · 10−5. Afterreaching its peak value, the learning rate is lin-early decayed. For RoBERTa, we found training tobe unstable with these parameters, so we chose alower learning rate of 1 ·10−5 and performed linearwarmup for the first 10,000 training steps.

To obtain results for our baselines on the rarifieddatasets, we use the original Mimick implementa-tion of Pinter et al. (2017), the A La Carte imple-mentation of Khodak et al. (2018) and the AttentiveMimicking implementation of Schick and Schutze(2019a) with their default hyperparameter settings.As A La Carte can only be used for words withat least one context, we keep the original BERTembeddings whenever no such context is available.

While using BERTRAM allows us to completelyremove the original BERT embeddings for all rarewords and still obtain improvements in accuracyon all three rarified downstream tasks, the same isnot true for RoBERTa, where removing the originalsequence of subword token embeddings for a givenrare word (i.e., not using the SLASH variant) hurtsperformance with accuracy dropping by 5.6, 7.4and 2.1 points for MNLI, AG’s News and DBPedia,respectively. We believe this to be due to the vastamount of additional contexts for rare words inRoBERTa’s training set that are not available toBERTRAM.