Composing and Embedding theWords-as-Classifiers Model of Grounded Semantics

Daniele MoroBoise State University

1910 University Dr.Boise, ID 83725danielemoro@

u.boisestate.edu

Stacy BlackBoise State University

1910 University Dr.Boise, ID 83725stacyblack@

u.boisestate.edu

Casey KenningtonBoise State University

1910 University Dr.Boise, ID 83725

caseykennington@boisestate.edu

Abstract

The words-as-classifiers model of groundedlexical semantics learns a semantic fitnessscore between physical entities and the wordsthat are used to denote those entities. In thispaper, we explore how such a model can in-crementally perform composition and how themodel can be unified with a distributional rep-resentation. For the latter, we leverage theclassifier coefficients as an embedding. Forcomposition, we leverage the underlying me-chanics of three different classifier types (i.e.,logistic regression, decision trees, and multi-layer perceptrons) to arrive at a several sys-tematic approaches to composition unique toeach classifier including both denotational andconnotational methods of composition. Wecompare these approaches to each other andto prior work in a visual reference resolutiontask using the refCOCO dataset. Our resultsdemonstrate the need to expand upon exist-ing composition strategies and bring togethergrounded and distributional representations.

1 Introduction

The words-as-classifiers (WAC) model of lexicalsemantics has shown promise as a way to ac-quire grounded word meanings with limited train-ing data in interactive settings. Introduced inKennington and Schlangen (2015) for groundingwords to visual aspects of objects, Schlangen et al.(2016) showed that the model could be general-ized to work with any object representation (e.g.,a layer in a convolutional neural network) with“real” objects depicted in photographs. The WAC

model builds on prior work (Larsson, 2015) treat-ing formal predicates as classifiers to effectivelylearn and determine class membership of entities(e.g., an object x denoted as “red” in an utterancebelongs to the predicate red(x) class). The WAC

model has been used to ground into modalitiesbeyond just vision, including simulated robotic

hand muscle activations (Moro and Kennington,2018), and WAC has been used for language un-derstanding in a fluid human-robot interaction ask(Hough and Schlangen, 2016) . Beyond compre-hension tasks, the WAC model has also been usedfor referring expression generation (Zarrieß andSchlangen, 2017). The WAC classifiers can use anyset of features from any modality and the model isinterpretable because each word has its own clas-sifier. Moreover, the WAC model allows for in-cremental, word-by-word composition which hasimplications for interactive dialogue: human usersof incremental spoken dialogue systems perceivethem as being more natural than non-incrementalsystems (Aist et al., 2006; Skantze and Schlangen,2009; Asri et al., 2014).

Though the WAC model has pleasing theo-retical properties and practical implications forinteractive dialogue and robotic tasks, Li andBoyer (2016) and Emerson and Copestake (2017)point out that WAC treats all words indepen-dently, thereby ignoring distributional relationsand meaning representations, and composition us-ing the WAC model has generally resulted in av-eraging over applications of WAC to objects–apurely intersectional approach to semantic com-position, which has been shown to fail in manycases (Kamp, 1975). As is well known, linguis-tic structures are compositional—simple elementscan be functionally combined into more complexelements (Frege, 1892)—and composition, whichis an important aspect of grounding into visual orother modalities, is itself a process that is a func-tion of how a particular model of lexical semanticsis represented.1

1To illustrate, following Baroni and Zamparelli (2010),the formal semantics approach to lexical semantics, as de-rived from Montague (1970, 1973), treats lexical meaning aspredicates which functionally determine class membership.In contrast, distributional representations (e.g., vector space

arX

iv:1

911.

0328

3v1

[cs

.CL

] 8

Nov

201

9

word1 word2 word3

Compose

Apply

word1 word2 word3

Compose

Apply Apply Apply

Figure 1: Left: connotational compose-then-apply;Right: denotational apply-then-compose.

The goal of this paper is (1) to explore and un-derstand possible approaches to composition forWAC, and (2) by determining the best WAC model,we extend WAC by using classifier coefficients asembedding vectors in a semantic similarity task.At the outset, following Schlangen et al. (2016),we make an important distinction by identifyingtwo composition types: in any task, compositioncan be handled at the level of connotation wherewords are composed then applied in the task (e.g.,tensor operations of word embeddings are gener-ally composed by summation before they are usedin a task, such as sentiment classification), and atthe level of denotation where words are appliedthen composed in a task (e.g., WAC classifiers areapplied to object representations, then the resultsof those applications are composed, such as ref-erence resolution to visual objects). These differ-ences in composition process are depicted in Fig-ure 1 which distinguishes between application to atask and composition.

More specifically, we alter the WAC model toleverage several classifier types, namely logisticregression, multi-layer perceptrons, and decisiontrees which allow us to use both connotative anddenotative composition strategies and benefit fromthe different classifiers, for example by graftingdecision trees together and by making use of thehidden layers in the multi-layer perceptrons (ex-plained in Section 4). Our evaluations show thatthe choice of classifier and the composition strate-gies that those classifiers afford yield varied, yetcomparable results in a visual reference resolutiontask (Section 5). Our analyses of the multi-layerperceptron model shows that the coefficients of theneurons in the hidden layers show properties thatare similar to distributional embedding approaches

models (Turney and Pantel, 2010) including embeddings) dif-fer from formal semantics in that word meanings are repre-sented not as predicates, but as high-dimensional vectors andas a result approaches to composition revolve around tensoroperations from summation and multiplication (Mitchell andLapata, 2010) to treating adjectives as matrices and nouns asvectors (Baroni and Zamparelli, 2010).

to lexical semantics (Section 5.3), which we eval-uate in Experiment 3. Finally, we conclude andexplain our plans for future work.

2 Related Work

Emerson and Copestake (2017) offer a compre-hensive review and discussion on different ap-proaches to compositionality including formalframeworks, tensor-based composition, and syn-tactic dependencies, to which we refer the reader.Their own model’s application of compositionuses distributional semantics and a probabilisticgraphical model.

Recursive neural networks can arguably encodecompositional processes directly, but it has beenshown recently in Lake and Baroni (2017) thatthey fail in a proof-of-concept neural machinetranslation task. The authors suggest a lack of sys-tematicity in how these networks learn the com-position process. Moreover, Socher et al. (2014)reported successful application of composition us-ing recursive neural models, but their approachmade use of syntactic information from depen-dency parses; i.e., the composition was likelyguided by the syntactic representation rather thanthe neural network. Yu et al. (2018) introduceMAttNet, which leverages recursive networks toobtain high results on a referring expression taskusing the same data we use for our experiments.In contrast with this work, our primary purpose isnot to achieve state-of-the-art results, but rather asystematic check of composition strategies.

Comparable to their work and ours here is Pa-perno et al. (2014) which attempted distributionalcomposition using linguistic information similarto dependency parses. These approaches and tasksgenerally apply a connotative strategy of compo-sition, where the final task is applied after com-bining meaning representations of the parts to firstarrive at a single meaning representation of a sen-tence. Our work also compares to Wu et al.(2019) which proposed a model that incorporatesgrounded and distributional information, whichwe explore in Experiment 3.

We extend prior work to mitigate WAC’s short-comings by modeling several different denotativeand connotative strategies made possible by themachinery of our chosen classifiers, and we usethe resulting classifier coefficients as a possibleway to bring WAC into a distributional space.2

2While we are not claiming that our approach is cogni-

1

Figure 2: Example of refCOCO image with three refer-ring expressions (one on each line) made to an imageregion (see Footnote 3 for source).

3 Data

To give context to understanding our model andcomposition approaches, we first explain the datathat we use in our experiments. We use thesame dataset as described in Schlangen et al.(2016), the “Microsoft Common Objects in Con-text” (MSCOCO) collection (Lin et al., 2014),which contains over 300k images with object seg-mentations, object labels, and image captions,augmented by Mao et al. (2016) to add Englishreferring expressions to image regions (i.e., ref-COCO). The average length of the referring ex-pressions is 8.3 tokens and the average number ofimage regions (which we treat as reference can-didates in our experiments) is 8. We used thedata access interface provided by Kazemzadehet al. (2014), which included a defined train-ing/valiation/test split of the data.3 An exampleimage, image region, and three referring expres-sions to that image region are depicted in Figure 2.

4 Model

The WAC approach to lexical semantics is es-sentially a task-independent approach to predict-ing semantic appropriateness of words in physical

tively plausible, we take inspiration from Barsalou (2015)which notes that physical situations play central roles inestablishing and using concepts–the WAC model is such agrounded model that considers physical context. Moreover,the WAC model is not an idealized account of concept rep-resentation; indeed, we take the vocabulary as our (verynoisy) ontology and the classifiers learn probabilistic map-pings which are far from idealized formal representations.

3https://github.com/lichengunc/refer

contexts. The WAC model pairs each word w in itsvocabulary V with a classifier that maps the real-valued features x of an entity ent to a semanticappropriateness (i.e., class membership) score:

[[w]]ent = λx.pw(x) (1)

For example, to learn the connotative meaningof the word red, the low-level features (e.g., vi-sual) of all objects referred to with the word red ina corpus of referring expressions are given as pos-itive instances to a supervised learning classifier.Negative instances are randomly sampled from thecomplementary set of referring expressions (i.e.,not containing the word red). This results in atrained λx.pred(x), where x is an object that canbe applied to red to determine class membership.

4.1 Approaches to Composition with WACTraditionally, the WAC model has been applied us-ing independent linear classifiers, such as logisticregression. In this paper, we expand upon the pre-vious work by conducting experiments with a va-riety of classifiers, such as multi-layer perceptrons(MLP) and decision trees (DT), in addition to lo-gistic regression (LR). We use these classifiers soas to not make assumptions of linearity (i.e., MLP)and to use the available machinery that MLPs andDTs afford us for exploring composition strategies.We explain below each approach and the methodsof composition for each. It is important to notethat in our explanations and discussions of compo-sition, we are focusing on application of the classi-fiers; all classifiers are trained as explained above(i.e., by pairing words with positive and negativeexamples). We leave more specific explanation oftraining to Section 5.

4.1.1 Logistic RegressionLR summed-predictions The traditional ap-proach to composition using WAC uses a denota-tive strategy where each WAC classifier in a re-ferring expression is applied to an entity (e.g., avisually present object in a scene) which yieldsa probability (i.e., a score of fitness) for eachword applied to each object. Following Schlangenet al. (2016), the resulting probabilities can becombined in several ways including summing, av-eraging, or multiplication (we opt for summingin our experiments) to produce a single overallexpression-level fitness score for each object. Thisoperation constitutes the composition of the refer-ring expression into a single distribution (hence,

denotative: the objects are applied to each wordclassifier, then the resulting probabilities are com-posed into a single distribution over candidate ob-jects). The object with the highest score is the hy-pothesized referred target.

4.1.2 Multi-layer PerceptronsWe explain in this section how we leverage MLPsfor three different approaches to composition. Un-like LR, the MLP does not need to assume linear-ity, and its structure allows us greater flexibility forcompositional techniques.

MLP summed-predictions For completenessand direct comparison to LR, we leverage MLPsas we did with the LR denotative approach by sim-ply using a MLP (i.e., using a single hidden layerof 3 neurons) in place of a LR classifier for eachword. Composition is applied after application bysumming the resulting probabilities.

Adjectives and Nouns Following Baroni andZamparelli (2010), we looked at adjective-nounpairs, which are two-word compositions that canidentify how an adjective qualifies a noun. Forexample, instead of treating large, green and treeseparately in the phrase the large green tree next tothe lake, we use the adjectives large and green tomodify the noun tree. We explore two approachesto composing adj-noun pairs using MLPs:

MLP adj-noun extended hidden layers: if thereexist multiple adjectives for one noun, a separateadj-noun pair is created for each adjective preced-ing the noun. When predicting, the MLP classi-fiers for each word in every adj-noun pair in thereferring expression is merged together into oneclassifier by extending the hidden layer to includeneurons from both the adjective and the noun in asingle MLP. The coefficients of the top layer (i.e., abinary sigmoid) of the original adj and noun MLPsare averaged together to produce a single proba-bility. The rest of the phrase is then composednormally using the traditional WAC methodology.This approach effectively leverages a connotativestrategy to compose the adj-noun pairs, then a de-notative strategy to compose the rest of the expres-sion.4

MLP adj-noun warm start: MLP machinery alsoaffords the use of the warm-start feature for the

4This differs from the LR connotative approach (i.e.,where the coefficients were averaged) because in this casethe coefficients in the hidden layer are not averaged, only thetop layer coefficients are averaged.

Figure 3: Example of decision tree graft of brown withdog: the root node of dog is grafted into the leaf nodesthat would output a “true” classification brown. Thiscomposes the noun-phrase “brown dog” (the dog clas-sifier augments the brown classifier).

MLPs. In this approach, we take the noun’s classi-fier in the adj-noun pair which has already beentrained, and then continue to train the classifier(i.e., using the warm-start functionality) with datathat was used to train the adjective’s classifier.This results in a single classifier that theoreticallyrepresents the entire adj-noun pair in a single clas-sifier. This approach keeps the classifiers the samesize, while leveraging a transfer-style learning ap-proach to produce a adj-noun pair classifier that iscomposed of its constituent words.

MLP extended hidden layers For this approach,we generalize the MLP adj-noun extended hiddenlayers approach and apply it to the entire expres-sion by concatenating the neurons in each wordMLP’s hidden layer and averaging the coefficientsof the top layer to form a single, composed MLP

that has a single hidden layer with nodes that candetermine fitness between objects and all words inan expression.

4.1.3 Decision TreesWe opted to apply DTs because of their readableinternal structure and how the branching systemlends itself to intuitive composition strategies.

DT summed-predictions For completeness anddirect comparison, we leverage DTs as we did withthe LR and MLP denotative approaches by simplyusing a DT classifier for each word. Compositionhappens after application by summing the result-ing predicted probabilities.

DT grafting In this approach to composition, weleverage the DT representation by training the clas-sifier as usual (i.e., independently), but then tocompose the classifiers into a single classifier, we“graft” the trees of each classifier wi in a refer-

ring expression to the two most probable “true”leaf nodes of the classifier wi−1, beginning withw1 as the starting tree with the root node. Thisgrafting process of two words is depicted in Fig-ure 3 where the root node of the decision tree forthe word dog is grafted into the true leaf nodesof the decision tree classifier for the word brown,thus allowing each word to make a contribution ofthe final decision of expression-level fitness to ob-jects. This grafting process is repeated for eachword in the referring expression, thus allowing usto apply then compose an entire phrase.

4.2 Composing Relational ExpressionsFor this final model, we follow and extend Ken-nington and Schlangen (2015)’s approach to re-lational expressions (e.g., containing prepositionssuch as next to or above, etc.) by expressing thoserelational phrases (r) as a fitness for pairs of ob-jects (i.e., R1 and R2; whereas for non-relationalwords, WAC only learns fitness scores for singleobjects):

[[r]]W = P (R1, R2|r) (2)

However, in prior work, the two candidate ob-jects were known. In our model, we only assumethat the final referred target object is known (i.e.,R1), but the relative object is not (i.e., R2). Tohandle this, we interpret relational phrases r con-taining prepositional words as “transitions” fromone noun phrase (NP1) to the relative noun phrase(NP2). For example, in the phrase the woman tothe right of the tree, we first identify the relationalphrase(s) (in this case right of ), and we use thisrelational phrase to make the most likely transi-tion from NP1 (i.e., the woman) to NP2 (i.e., thetree). We do this by using a learned WAC model(i.e., trained on single objects using any of the ap-proaches explained above) and apply NP2 to allcandidate objects in the scene. This produces adistribution over all objects (i.e., a partially ob-servableNP2); we use the resulting argmax of thatdistribution to arrive at the R2 that was referred toby NP2. With R1 and R2, we train r by takingthe difference in features (i.e., simple vector sub-traction) between the feature vector for R1 and R2

resulting in a feature vector for r that is the samesize as for all other WAC classifiers. During ap-plication, we find the most likely pair by applyingNP1 and NP2 to all objects, then r to all pairs ofobjects, and force identity on theNP1 distributionand candidate R1, as well as the NP2 distribution

and the candidate R2, forming a trellis-like struc-ture. The combined (i.e., product) of probabilitiesfor NP1, r, and NP2 result in a final distribution;we take the argmax probability and the resultingR1 object as the target referred object.

5 Experiments

5.1 Experiment 1: Simple Expressions

In this experiment, we evaluate the performanceof our approaches to composition of simple (i.e.,containing no relational words) referring expres-sion resolution using the refCOCO data.

Task & Procedure The task is reference resolu-tion to objects depicted in static images. For ex-ample, the referring expression the woman in redsitting on the left would require composition ofeach word (with the exception of the quantifier thewhich signals that the referring expression shouldonly identify a single entity). We task the WAC

model (and varying compositional approaches, asexplained above) to produce a “distribution” (i.e.,ranked scores) of each object in an image. Weonly considered referring expressions that did notcontain words included in this list: below, above,between, not, behind, under, underneath, front of,right of, left of, ontop of, next to, middle of (aswas done in Schlangen et al. (2016)) resulting insimpler referring expressions. This resulted in106,336 training instances and 9,304 test instances(i.e., individual referring expressions and the cor-responding image with multiple object regions).In Experiment 2 we consider all referring expres-sions, including those with relations.

Training the WAC model follows prior workwhere each word in the refCOCO training datais trained on all instances where that word wasused to refer to an object and 5 randomly chosennegative examples sampled from the training datawhere that particular word was not used in a re-ferring expression (i.e., a 5-to-1 negative to pos-itive example ratio). For each object in the im-ages, following Schlangen et al. (2016), we usedthe image region information from the annotateddata and extract that region as a separate imagethat we then pass through GoogLeNet (Szegedyet al., 2015), a convolutional neural network thatwas trained on data from the Large Scale VisualRecognition Challenge 2014 (ILSVRC2014) fromthe ImageNet corpus (Jia Deng et al., 2009). Thatis, GoogLeNet is optimized to recognize single

objects within an image, which it can do effec-tively with our extracted image regions. This re-sults in a vector representation of the object (i.e.,region) with 1024 dimensions (i.e., we use thelayer directly below the predictions layer).5 Weconcatenate to this vector 7 additional features thatgive information about the image region resultingin a vector of 1031 dimensions: the (relative tothe full image from which it was extracted) coor-dinates of two corners, its (relative) area, distanceto the center, and orientation of the image.

To ensure that our WAC models were made upof reliable classifiers, we threw out words that had4 or fewer positive training examples, resulting inan English vocabulary of 2,349. For training theLR WAC model, we used L1 normalization. Fortraining the DT WAC, we used the GINI splittingcriteria with a maximum depth of 2. For trainingthe MLP WAC model, we used a multi-layer per-ceptron with a single hidden layer of 3 neuronsusing the tanh activation function, the top layer isa binary sigmoid, trained using the adam solver(alpha value of 0.1) for 2000 maximum epochs.In all cases, the best hyper parameters were foundusing the development data.

Metrics The metric we use for this task is ac-curacy that the highest scoring object in an im-age of candidate objects as ranked by our modelmatches the annotated target object referred to bythe referring expression. For example, in Figure 2,there are several annotated objects including thetwo people, the trees, car, ground, etc., but thereferring expression woman on the right in whiteshirt should uniquely identify the target object inthe annotated image region. Our target is 0.64, theresult reported in Schlangen et al. (2016).

Results The results for this experiment are in theExp. 1 acc column of Table 1.6 We see that inall cases, the denotative summed-predictions forall three classifiers yield respectable performance(the first row verifies results reported in Schlangenet al. (2016) for this task). When considering con-notative approaches, the results are more nuanced:for the MLP adj-noun approaches, we see similar

5We used the development set to evaluate other existingneural networks trained on the ImageNet data, including ap-proaches more recent and with better performance on the taskthan GoogLeNet, but we found that GoogLeNet worked bet-ter for our task.

6Yu et al. (2018) yielded far better results than we showhere–our goal is to explore compositionality directly using amore transparent model.

model Exp. 1 acc Exp. 2 accLR summed-preds 0.64 0.62MLP summed-preds 0.63 0.61MLP adj-noun ext. hidden 0.62 0.60MLP adj-noun warm-start 0.62 0.61MLP ext. hidden 0.6 0.58MLP relational 0.62 0.63DT summed-preds 0.55 0.54DT graft 0.39 0.27

Table 1: Experiment 1 & 2 results: accuracy scoresfor the models and composition approaches using therefCOCO data.

results when the hidden layers are extended andwhen we use warm-start. This is a positive re-sult in that composing adj-noun pairs together us-ing warm-start is theoretically appealing becausea single classifier can perform the work of two,though it needs additional training data to per-form that function, whereas extending the hiddenlayers requires no additional retraining, and thecompose-then-apply nature of it makes it more ap-pealing than simply summing the predictions ofeach word applied to the objects, as has been donein prior work. Unfortunately, when extending thehidden layers to contain the neurons from the MLP

classifiers for all words in the expression, the re-sults take a hit when compared to the denotativesummed predictions approach. The DT classifiersdid not perform well in this particular task (boththe denotative summed predictions and the graft-ing) which is somewhat surprising, though as canbe seen in Figure 3, by limiting the depth, the DT

classifiers were unable to make use of the nuancein the individual features.

5.2 Experiment 2: Referring Expressionswith Relations

For this experiment, we apply our approaches ofcomposition to all of the refCOCO data, includingcases where relations are present.

Task & Procedure The task and procedure forthis experiment are similar to that of Experiment1 with the important difference that we considerall of the training and test instances (120,266 and9,914, respectively) for training; we do not ignorereferring expressions that have relations, for ex-ample the woman in red sitting on the left next tothe tree.

Metrics The metrics for this experiment are thesame as Experiment 1: accuracy that the highestscoring object as ranked by our model matches the

annotated target object referred to by the referringexpression.

Results Table 1, column Exp. 2 acc contains theresults for this experiment. As expected, nearly allapproaches took a hit because they were unable tohandle the composition of multiple noun phrasesand learn the semantics of relational words. Thebest performing model, as expected, is the MLP re-lational model which applies the extended hiddenlayer to individual noun phrases, but composesthem using the relational word using summedpredictions (the results are statistically significantcompared to the LR summed-preds row result forExp 2). The final relational model, which uses theextended hidden approach for each noun phrase,applies more principled connotative compositionand works with relational expressions without re-quiring annotation of the relative object.

5.3 AnalysesWe end this experiment by offering some analysison what the WAC classifiers are learning and com-pare some of the composition strategies with eachother. We put focus on the MLP classifier since itproduced the best results and allows more possiblecomposition strategies.

Individual Classifiers Similar to Kenningtonand Schlangen (2015), we looked at classifiersthat, we assume, learned features relating to col-ors. To check this, we ranged over a wide rangeof colors, producing an image for each color, andpassed those through the GoogLeNet to arrive at arepresentation for each color that our model coulduse. Our analyses show that the classifiers learnedprototypical colors for objects as well as colorterms. For example, we passed all colors throughthe water classifier and plotted the resulting prob-abilities, resulting in Figure 4. This shows thatindividual classifiers can pick up on color infor-mation where an important visual property is itscolor (i.e., water), though that information is notapparent in the underlying representation (i.e., theGoogLeNet vector).

This is an important result with useful im-plications: this kind of transfer learning usingGoogLeNet (or any other model) trained on theImageNet data extends the original limited abilityof those networks by using the entire vocabulary toidentify not only object types, but also attributes(e.g., colors, sizes, spatial placements, etc.). Im-portantly for interactive dialogue with robots, the

Figure 4: Probabilities returned by passing color im-ages through the water classifier from the red to theviolet range.

WAC approach allows new vocabulary to be addedword-by-word without requiring retraining of theentire underlying network.

Semantic Clusters To see if WAC could alsoyield semantic clusters, we applied a t-distributedStochastic Neighbor Embedding (TSNE) (van derMaaten and Hinton, 2008) to the hidden layersof the MLP classifiers (i.e., 3021 dimensions, aseach hidden layer had 3 neurons, each neuron hadcoefficients for 1007 features), mapping the coef-ficient vectors to 2 dimensions, then applied theDensity-based Spatial Clustering of Applicationswith Noise (dbscan; eps = 0.7) (Schubert et al.,2017) to cluster the TSNE result, which is depictedin Figure 5.7 The figure shows a large, centralcluster with more informative clusters closer to theedges. The following are several noteworthy clus-ters:

Figure 5: Cluster proposals of MLP classifier hiddenlayer node coefficients.

• yellow, red, green, blue, light, board• area, between, of, above, edge, next to, right• door, oven, stove, dishwasher, knobs• trailer, semi, rig, car, vehicle, van, ups, suv, taxi• cat, dog, horse, cow, sheep, animal

7We found that logistic regression coefficients did not re-sult in any meaningful clusters.

This analysis informs us that using the MLP

classifier is not only more satisfying for compo-sition approaches, but it can also yield coefficientsthat can be used as vectors for word embeddings.Our final experiment tests this hypothesis.

5.4 Experiment 3: WAC Coefficients forSemantic Similarity

In this experiment, we evaluate the vectors that arederived from the coefficients of the MLP variant ofWAC in a semantic similarity task. We follow asimilar approach to Kottur et al. (2016) by creat-ing visually grounded word embeddings, but wemake use of the WAC model and we create ourown dataset for this experiment.

Task & Procedure We evaluated the WAC co-efficient vectors using the semantic similaritytasks WordSim-353 (Finkelstein et al., 2002) andSimLex-999 (Hill et al., 2015) by comparing theWAC vectors with GloVe embeddings (Penningtonet al., 2014) (6B, 200d), and a concatenation of thevectors for each word in the two models. To arriveat a model for WAC with a large enough vocabu-lary, we retrieved 100 images (i.e., using GoogleImage Search) for each of 30,000 common En-glish words and trained the MLP classifiers as de-scribed in Experiment 1 above using 100 imagesfor each word.

Metrics For semantic similarity, we report aSpearman Correlation between the cosine similar-ities of the WAC coefficient vector, the GloVe em-beddings, and the combination of the two models.High numbers denote better scores.

Results The results of the Spearman Correla-tion is shown in Table 2. The results show that,when combined with embeddings trained on largeamounts of text such as GloVe, the WAC coef-ficient vectors can contribute useful informationderived from the visual features that the GloVeembeddings don’t have. We conjecture that WAC

does not work well on its own because the wordsare trained independently, and the WAC model as-sumes that words trained on visual representationsare concrete, whereas many words are abstract, thesemantics of which is learned in lexical context,such as distributional embeddings. These resultsshow that WAC is a potential model for bridginggrounded and distributional approaches to lexicalsemantics, which we will explore further in futurework.

model WordSim-353 SimLex-999WAC 0.485 0.157GloVe 0.630 0.339combined 0.707 0.326

Table 2: Experiment 3 results: Spearman Correlationbetween the cosine similarities of WAC, GloVe, and thetwo combined.

6 Conclusions

In this paper, we explored using LR, MLP, and DT

for composing the words-as-classifiers approachto grounded, lexical semantics. We evaluated sev-eral methods including denotative where the clas-sifiers are applied to objects, then the resultingprobabilities are summed, connotative where thecomposition was applied first (i.e., either by ex-tending hidden layers or using warm start in MLP,or grafting DT). Our results show that classifiersaffect results, and the kinds of composition thatcan be accomplished varies depending on the clas-sifier. We concur with Baroni (2019) that more ex-ploration needs to be done in this area to learn thesystematic functional applications of compositionof language; our results are an important step inthat direction.

Furthermore, we showed that the WAC modelhas properties that lends itself for a straightfor-ward embedding by using the classifier coeffi-cients. We showed in Experiment 3 that the em-beddings show promise in a simple word similar-ity task. We are in the process of evaluating theWAC embeddings on other common tasks and an-alyzing the efficacy of WAC embeddings by com-bining them with other approaches to semantics.

Combined with prior work, the results reportedhere have implications for tasks and settings forspeech interaction: WAC can be trained with mini-mal training data, used in dialogue systems wherewords ground to multiple modalities (e.g., vision,proprioperception, predicted emotions, etc.), canbe composed incrementally, and the coefficientsof the classifiers can be unified with other distri-butional representations.

For future work, we will explore how WAC canbe coupled with formal and distributional seman-tic representations to better exploit and integrateknowledge from multiple modalities for a moreholistic representation of semantics. We are alsoevaluating WAC in an interactive language learn-ing task with two different robot platforms.

ReferencesGregory Aist, James Allen, Ellen Campana, Lucian

Galescu, Carlos Gallo, Scott Stoness, Mary Swift,and Michael Tanenhaus. 2006. Software archi-tectures for incremental understanding of humanspeech. In Proceedings of CSLP, pages 1922—-1925.

Layla El Asri, Romain Laroche, Olivier Pietquin, andHatim Khouzaimi. 2014. NASTIA: NegotiatingAppointment Setting Interface. In Proceedings ofLREC, pages 266–271.

Marco Baroni. 2019. Linguistic generalization andcompositionality in modern artificial neural net-works. arXiv.

Marco Baroni and Roberto Zamparelli. 2010. Nounsare vectors, adjectives are matrices: Representingadjective-noun constructions in semantic space. InProceedings of the 2010 Conference on . . . , October,pages 1183–1193, Cambridge, MA. Association forComputational Linguistics.

Lawrence W. Barsalou. 2015. Cognitively PlausibleTheories of Concept Composition. In Composi-tionality and concepts in linguistics and psychology,pages 1–31. Springer, Cham.

Guy Emerson and Ann Copestake. 2017. SemanticComposition via Probabilistic Model Theory. Pro-ceedings of IWCS.

Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias,Ehud Rivlin, Zach Solan, Gadi Wolfman, and Ey-tan Ruppin. 2002. Placing search in context: theconcept revisited. ACM Transactions on Informa-tion Systems, 20(1):116–131.

Gottlob Frege. 1892. Uber Sinn und Bedeutung.Erkenntnis, 100(1):1–15.

Felix Hill, Roi Reichart, and Anna Korhonen. 2015.Simlex-999: Evaluating semantic models with(Genuine) similarity estimation. ComputationalLinguistics.

Julian Hough and David Schlangen. 2016. Investigat-ing Fluidity for Human-Robot Interaction with Real-time, Real-world Grounding Strategies. In Proceed-ings of the 17th Annual Meeting of the Special Inter-est Group on Discourse and Dialogue, pages 288–298, Los Angeles. Association for ComputationalLinguistics.

Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, andLi Fei-Fei. 2009. ImageNet: A large-scale hierar-chical image database. In 2009 IEEE Conference onComputer Vision and Pattern Recognition.

Hans Kamp. 1975. Two Theories about Adjectives.In Formal Semantics of Natural Language, chapterTwo theori. Cambridge University Press.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten,and Tamara L Berg. 2014. ReferItGame: Refer-ring to Objects in Photographs of Natural Scenes.In Proceedings of the 2014 Conrerence on Em-pirical Methods in Natural Language Processing(EMNLP’14), pages 787–798.

C. Kennington and D. Schlangen. 2015. Simple learn-ing and compositional application of perceptuallygroundedword meanings for incremental referenceresolution. In ACL-IJCNLP 2015 - 53rd AnnualMeeting of the Association for Computational Lin-guistics and the 7th International Joint Conferenceon Natural Language Processing of the Asian Feder-ation of Natural Language Processing, Proceedingsof the Conference, volume 1.

Satwik Kottur, Ramakrishna Vedantam, Jose MFMoura, and Devi Parikh. 2016. Visual word2vec(vis-w2v): Learning visually grounded word embed-dings using abstract scenes. In Proceedings of theIEEE Conference on Computer Vision and PatternRecognition, pages 4985–4994.

Brenden M. Lake and Marco Baroni. 2017. General-ization without systematicity: On the compositionalskills of sequence-to-sequence recurrent networks.arXiv.

Staffan Larsson. 2015. Formal semantics for percep-tual classification. Journal of Logic and Computa-tion, 25(2):335–369.

Xiaolong Li and Kristy Elizabeth Boyer. 2016. Refer-ence Resolution in Situated Dialogue with LearnedSemantics. In Proceedings of Sigdial, August. As-sociation for Computational Linguistics.

Tsung-Yi Lin, Michael Maire, Serge Belongie, JamesHays, Pietro Perona, Deva Ramanan, Piotr Dollar,and C Lawrence Zitnick. 2014. Microsoft COCO:Common Objects in Context. In Computer Vision –ECCV 2014, pages 740–755, Cham. Springer Inter-national Publishing.

Laurens van der Maaten and Geoffrey Hinton. 2008.Visualizing Data using t-SNE. Journal of MachineLearning Research.

Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, ZhihengHuang, and Alan L. Yuille. 2016. Learning like achild: Fast novel visual concept learning from sen-tence descriptions of images. In Proceedings of theIEEE International Conference on Computer Vision,volume 11-18-Dece.

Jeff Mitchell and Mirella Lapata. 2010. Compositionin distributional models of semantics. Cognitive sci-ence, 34(8):1388–1429.

Richard Montague. 1970. Universal grammar. Theo-ria, 36(3):373–398.

Richard Montague. 1973. The proper treatment ofquantification in ordinary English. In Approachesto natural language, pages 221–242. Springer.

Daniele Moro and Casey Kennington. 2018. Multi-modal Visual and Simulated Muscle Activations forGrounded Semantics of Hand-related Descriptions.In Proceedings of the 22nd Workshop onthe Seman-tics and Pragmatics of Dialogue.

Denis Paperno, Nghia The Pham, and Marco Baroni.2014. A practical and linguistically-motivated ap-proach to compositional distributional semantics. InProceedings of ACL, pages 90–99. Association forComputational Linguistics.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. Glove: Global Vectors for WordRepresentation. In Proceedings of the 2014 Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 1532–1543.

David Schlangen, Sina Zarriess, and Casey Kenning-ton. 2016. Resolving References to Objects in Pho-tographs using the Words-As-Classifiers Model. InProceedings of the 54th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 1213–1223.

Erich Schubert, Martin Ester, Xiaowei Xu, Hans PeterKriegel, and Jorg Sander. 2017. DBSCAN Revis-ited, Revisited: Why and How You Should (Still)Use DBSCAN. ACM Transactions on DatabaseSystems.

Gabriel Skantze and David Schlangen. 2009. Incre-mental dialogue processing in a micro-domain. Pro-ceedings of the 12th Conference of the EuropeanChapter of the Association for Computational Lin-guistics on EACL 09, (April):745–753.

Richard Socher, Andrej Karpathy, Quoc V Le, Christo-pher D Manning, and Andrew Y Ng. 2014.

Grounded Compositional Semantics for Findingand Describing Images with Sentences. Tacl,2(April):207–218.

Christian Szegedy, Wei Liu, Yangqing Jia, PierreSermanet, Scott Reed, Dragomir Anguelov, Du-mitru Erhan, Vincent Vanhoucke, and Andrew Ra-binovich. 2015. Going deeper with convolutions. InProceedings of the IEEE Computer Society Confer-ence on Computer Vision and Pattern Recognition.

Peter D Turney and Patrick Pantel. 2010. From Fre-quency to Meaning: Vector Space Models of Seman-tics. Artificial Intelligence, 37(1):141–188.

Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang,Lei Li, Weiwei Sun, and Wei-Ying Ma. 2019. Uni-fied Visual-Semantic Embeddings: Bridging Visionand Language with Structured Meaning Representa-tions. In Proceedings of the Combined Workshop onSpatial Language Understanding (SpLU) GroundedCommunication for Robotics (RoboNLP) GroundedCommunication for Robotics (RoboNLP).

Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, XinLu, Mohit Bansal, and Tamara L Berg. 2018. Mat-tnet: Modular attention network for referring expres-sion comprehension. In Proceedings of the IEEEConference on Computer Vision and Pattern Recog-nition, pages 1307–1315.

Sina Zarrieß and David Schlangen. 2017. Is this achild, a girl or a car? Exploring the contribution ofdistributional similarity to learning referential wordmeanings. In Proceedings of the 15th Conference ofthe European Chapter of the Association for Com-putational Linguistics: Volume 2, Short Papers, vol-

ume 2, pages 86–91.