Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3975–3989August 1–6, 2021. ©2021 Association for Computational Linguistics

3975

Multi-Task Learning and Adapted Knowledge Models for Emotion-CauseExtraction

Elsbeth Turcan1∗

Kasturi Bhattacharjee3Shuai Wang3

Yaser Al-Onaizan3

1Department of Computer Science, Columbia University2Data Science Institute, Columbia University

3Amazon AI{eturcan, smara}@cs.columbia.edu

{wshui, ranubhai, kastb, onaizan}@amazon.com

Rishita Anubhai3Smaranda Muresan2,3

Abstract

Detecting what emotions are expressed intext is a well-studied problem in natural lan-guage processing. However, research on finer-grained emotion analysis such as what causesan emotion is still in its infancy. We presentsolutions that tackle both emotion recognitionand emotion cause detection in a joint fash-ion. Considering that common-sense knowl-edge plays an important role in understandingimplicitly expressed emotions and the reasonsfor those emotions, we propose novel meth-ods that combine common-sense knowledgevia adapted knowledge models with multi-tasklearning to perform joint emotion classifica-tion and emotion cause tagging. We show per-formance improvement on both tasks when in-cluding common-sense reasoning and a multi-task framework. We provide a thorough analy-sis to gain insights into model performance.

1 Introduction

Utterance and document level emotion recognitionhas received significant attention from the researchcommunity (Mohammad et al., 2018; Poria et al.,2020a). Given the utterance Sudan protests: Out-rage as troops open fire on protestors an emotionrecognition system will be able to detect that angeris the main expressed emotion, signaled by theword "outrage". However, the semantic informa-tion associated with expressions of emotion, suchas the cause (the thing that triggers the emotion)or the target (the thing toward which the emotionis directed), is important to provide a finer-grainedunderstanding of the text that might be needed inreal-world applications. In the above utterance, thecause of the anger emotion is the event “troopsopen fire on protestors”, while the target is the en-tity "troops" (see Figure 1) .

∗Work done during an internship with Amazon AI.

Research on finer-grained emotion analysis suchas detecting the cause for an emotion expressed intext is in its infancy. Most work on emotion-causedetection has utilized a Chinese dataset where thecause is always syntactically realized as a clauseand thus was modeled as a classification task (Guiet al., 2016). However, recently Bostan et al. (2020)and Oberländer and Klinger (2020) argued thatin English, an emotion cause can be expressedsyntactically as a clause (as troops open fire onprotestors), noun phrase (1,000 non-perishablefood donations) or verb phrase (jumped into anice-cold river), and thus we follow their approachof framing emotion cause detection as a sequencetagging task.

We propose several ways in which to approachthe tasks of emotion recognition and emotion causetagging. First, these two tasks should not be in-dependent; because the cause is the trigger forthe emotion, knowledge about what the cause isshould narrow down what emotion may be ex-pressed, and vice versa. Therefore, we presenta multi-task learning framework to model themjointly. Second, considering that common-senseknowledge plays an important role in understand-ing implicitly expressed emotions and the reasonsfor those emotions, we explore the use of common-sense knowledge via adapted knowledge models(COMET, Bosselut et al. (2019)) for both tasks. Akey feature of our approach is to combine theseadapted knowledge models (i.e., COMET), whichare specifically trained to use and express common-sense knowledge, with pre-trained language mod-els such as BERT, (Devlin et al., 2019).

Our primary contributions are three-fold: (i) anunder-studied formulation of the emotion cause de-tection problem as a sequence tagging problem; (ii)a set of models that perform the emotion classifica-tion and emotion cause tagging tasks jointly while

3976

using common-sense knowledge (subsection 4.2)with improved performance (section 6); and (iii)analysis to gain insight into both model perfor-mance and the GoodNewsEveryone dataset thatwe use (Bostan et al., 2020) (section 7).

2 Related Work

Emotion detection is a widely studied subfield ofnatural language processing (Mohammad et al.,2018; Poria et al., 2020a), and has been appliedto a variety of text genres such as fictional stories(Alm et al., 2005), news headlines (Strapparavaand Mihalcea, 2010), and social media, especiallymicroblogs such as Twitter (Abdul-Mageed andUngar, 2017; Kiritchenko et al., 2014; Rosenthalet al., 2019; Mohammad et al., 2018). Earlier work,including some of the above, focused on feature-based machine learning models that could leverageemotion lexicons (Mohammad and Turney, 2013)),while recent work explores deep learning models(e.g., Bi-LSTM and BERT) and multi-task learning(Xu et al., 2018; Demszky et al., 2020).

However, comparatively few researchers havelooked at the semantic roles related to emotion suchas the cause, the target or the experiencer, with fewexceptions for Chinese (Gui et al., 2016; Chen et al.,2018; Xia and Ding, 2019; Xia et al., 2019; Fanet al., 2020; Wei et al., 2020; Ding et al., 2020), En-glish (Mohammad et al., 2014; Ghazi et al., 2015;Kim and Klinger, 2018; Bostan et al., 2020; Ober-länder et al., 2020; Oberländer and Klinger, 2020)and Italian (Russo et al., 2011). We highlight someof these works here and draw connection to ourwork. Most recent work on emotion-cause detec-tion has been carried out on a Chinese dataset com-piled by Gui et al. (2016). This dataset character-izes the emotion and cause detection problems asclause-level pair extraction problem – i.e., of all theclauses in the input, one is selected to contain theexpression of an emotion, and one or more (usuallyone) are selected to contain the cause of that emo-tion. Many publications have used this corpus todevelop novel and effective model architectures forthe clause-level classification problem (Chen et al.,2018; Xia and Ding, 2019; Xia et al., 2019; Fanet al., 2020; Wei et al., 2020; Ding et al., 2020). Thekey difference between this work and ours is thatwe perform cause detection as a sequence-taggingproblem: the cause may appear anywhere in theinput, and may be expressed as any grammaticalconstruction (a noun phrase, a verb phrase, or a

Figure 1: An example of the semantic roles annotatedby Bostan et al. (2020)

clause). Moreover, we use common sense knowl-edge for both tasks (emotion and cause tagging),through the use of adapted language models suchas COMET.

For English, several datasets have been intro-duced (Mohammad et al., 2014; Kim and Klinger,2018; Ghazi et al., 2015; Bostan et al., 2020; Po-ria et al., 2020b), and emotion cause detection hasbeen tackled either as a classification problem (Mo-hammad et al., 2014), or as a sequence tagging orspan detection problem (Kim and Klinger, 2018;Ghazi et al., 2015; Oberländer and Klinger, 2020;Poria et al., 2020b). We particularly note the workof Oberländer and Klinger (2020), who argue forour problem formulation of cause detection as se-quence tagging rather than as a classification tasksupported by empirical evidence on several datasetsincluding the GoodNewsEveryone dataset (Bostanet al., 2020) we use in this paper. One contributionwe bring compared to these models is that we for-mulate a multi-task learning framework to jointlylearn the emotion and the cause span. Anothercontribution is the use of common-sense knowl-edge through the use of adapted knowledge modelssuch as COMET (both in the single models and themulti-task models). Ghosal et al. (2020) have veryrecently shown the usefulness of common-sensereasoning to the task of conversational emotiondetection.

3 Data

For our experiments, we use the GoodNewsEvery-one corpus (Bostan et al., 2020), which contains5,000 news headlines labeled with emotions andsemantic roles such as the target, experiencer, andcause of the emotion, as shown in Figure 1.1 Wefocus on the emotion detection and cause taggingtasks in this work. To our knowledge, GoodNew-sEveryone is the largest English dataset labeled for

1While the dataset labels both the most dominant emotionexpressed in text and the reader’s emotion, for this paper weonly focus on the former.

3977

Figure 2: Distribution of adjudicated emotion labels inthe GoodNewsEveryone train data, as a percentage ofthe data points. “Positive” and “Negative” are abbrevi-ated as + and -.

both of these tasks.In our experiments, we limit ourselves to the

data points for which a cause span was annotated(4,798). We also note that this dataset uses a 15-way emotion classification scheme, an extended setincluding the eight basic Plutchik emotions as wellas additional emotions like shame and optimism.While a more fine-grained label set is useful for cap-turing subtle nuances of emotion, many externalresources focus on a smaller set of emotions. Wealso note that the label distribution of this datasetheavily favors the more basic emotions, as shownin Figure 2. Therefore, for our work, we choose tolimit ourselves to the six Ekman emotions (anger,fear, disgust, joy, surprise, and sadness). We alsochoose to keep positive surprise and negative sur-prise separated, to avoid severely unbalancing thelabel distribution for our experiments. We ran-domly split the remaining data (2,503 data points)into 80% train, 10% development, and 10% test.

4 Models

An important feature showcased by the Good-NewsEveryone dataset is that causes of emotionscan be expressed through different syntactic con-stituents such as clauses, verb phrases, or noun-phrases. Thus, we approach the cause detectionproblem as a sequence tagging problem using theIOB scheme (Ramshaw and Marcus, 1995): C ={I-cause,O,B-cause}. Our approach is supportedby very recent results by Oberländer and Klinger(2020) and Yuan et al. (2020) who show that model-ing emotion cause detection as a sequence taggingproblem is better suited than a clause classification

problem, although not much current work has yetadopted this formulation. We tackle the emotion de-tection task as a seven-way classification task withE = {anger, disgust, fear, joy, sadness, negativesurprise, positive surprise}.

4.1 Single-Task Models

As a baseline, we train single-task models foreach of emotion classification and cause span tag-ging. We use a pre-trained BERT language model2

(Devlin et al., 2019), which we fine-tune on ourdata, as the basis of this model. Our prepro-cessing strategy for all of our models consistsof the pretrained BERT vocabulary and Word-Piece tokenizer3 (Wu et al., 2016) from Hug-gingface (Wolf et al., 2020). Therefore, for asequence of n WordPiece tokens, our input tothe BERT model is a sequence of n + 2 tokens,X = [[CLS], x1, x2, ...xn, [SEP]], where each xiis from a finite WordPiece vocabulary and [CLS]and [SEP] are BERT’s begin and end tokens. Pass-ing X through BERT yields a sequence of vectorhidden states H = [h[CLS], h1, h2, ..., hn, h[SEP ]]with dimension dBERT = 768. For emotion classi-fication, we pool these hidden states and allow hy-perparameter tuning to select the best type: select-ing the [CLS] token (hf = h[CLS]), mean pooling

(hf =∑n

i=1 hi

n ), max pooling (hf,j = maxhi,j), orattention as formulated by Bahdanau et al. (2015):

hf =

n∑i=1

αihi (1)

where αi = exp (Wahi+ba)∑nj=1 exp (Wahj+ba)

for trainable

weights Wa ∈ R1×dBERT and ba ∈ R1. Then, thefinal distribution of emotion scores is calculated bya single dense layer and a softmax:

e = softmax(Wehf + be) (2)

with e ∈ R|E| and for trainable parametersWe ∈R|E|×dBERT and be ∈ R|E|. For cause tagging, atag probability distribution is calculated directly oneach hidden state:

ci = softmax(Wchi + bc) (3)

2We use BERT-BASE-UNCASED. We experimented withBERT-BASE-CASED, but it underperformed as the headlinesincorporated into GoodNewsEveryone come from differentnews sources and have different capitalization styles.

3In the tagging setting, we ignore all tags predicted forsubword tokens and use only the tag of the first subword.

3978

(a) The MultiC�E model. (b) The MultiE�C model.

Figure 3: Our multi-task models.

with ci ∈ R|C| and for trainable parametersWc ∈ R|C|×dBERT and bc ∈ R|C|. We refer toboth of these single-task models as BERT; if thetask is not clear from the context, we will refer tothe emotion detection model as BERTE and thecause tagging model as BERTC . Our training lossfor emotion classification as well as emotion causetagging is the mean negative log-likelihood (NLL)loss per minibatch of size b:

NLLemo = −1

b

∑j

∑k

yjk log ejk (4)

NLLcause = −1

b

∑i

∑j

∑k

yijk log cijk (5)

where j is the index of the sentence in the mini-batch, k is the index of the label being consid-ered (emotion labels for NLLemo and IOB tags forNLLcause), i is the index of the ith token in the jth

sentence in the minibatch, yjk ∈ {0, 1} is the goldprobability of the kth emotion label for the jth sen-tence, yijk ∈ {0, 1} is the gold probability of thekth cause tag for the ith token in the jth sentence,and ejk and cijk are the output probabilities of thekth emotion label and of the kth cause label for theith token, both for the jth sentence.

4.2 Multi-Task Models

Our hypothesis is that the emotion detection andcause tagging tasks are closely related and can in-form each other; therefore we propose three multi-task learning models to test this hypothesis. For allmulti-task models, we use the same base architec-ture (BERT) as the single models. Additionally, forthese models, we combine the losses of both tasksand weight them with a tunable lambda parameter:

λNLLemo + (1 − λ)NLLcause, using NLLemo andNLLcause from Equation 4 and Equation 5.

Multi. The first model, Multi, is the classicalmulti-task learning framework with hard parametersharing, where both tasks share the same BERTlayers. Two dense layers for emotion classificationand cause tagging operate at the same time fromthe same BERT layers, and we train both of thetasks simultaneously. That is, we simply calculateour emotion scores e and cause tag scores c fromthe same set of hidden states H .

We further develop two additional multi-taskmodels with the intuition that we can design moreexplicit and concrete task dependencies than simpleparameter sharing in the representation layer.

MultiC�E . We assume that if a certain text spanis given as the cause of an emotion, it should bepossible to classify that emotion correctly whilelooking only at the words of the cause span. There-fore, we propose the MultiC�E model, the architec-ture of which is illustrated in Figure 3a. This modelbegins with the single-task cause detection model,BERTC , which produces a probability distributionP (yi|xi) over IOB tags for each token xi, whereP (yi|xi) = ci from Equation 3. Then, for eachtoken, we calculate the probability that it is part ofthe cause as P (Cause|xi) = P (B|xi)+P (I|xi) =1 − P (O|xi). We feed the resulting probabilitiesthrough a softmax over the sequence and use themas an attention distribution over the input tokens inorder to pool the hidden representations and per-form emotion classification: attention is computedas in Equation 1, where αi =

expP (Cause|xi)∑nj=1 expP (Cause|xi)

,and emotion classification as in Equation 2. Forthe MultiC�E model, we apply teacher forcing attraining time, and the gold cause spans are used to

3979

Figure 4: The architecture of our proposedMultiCOMET

C�E model.

create the attention weights before emotion classi-fication (which means that P (Cause|xi) ∈ {0, 1}).At inference time, the model uses the predictedcause span instead.

MultiE�C . Next, we hypothesize that knowledgeof the predicted emotion should help us identifysalient cause words. The MultiE�C model firstperforms emotion classification, which results ina probability distribution over predicted emotionlabels, as in the BERTE model and Equation 2. Weadditionally keep an emotion embedding matrixE, where E[i] is a learnable representation of thei-th emotion label (see Figure 3b) with dimensionde (in our experiments, we set de = 300). Weuse the predicted label probabilities e to calculatea weighted sum of the emotion embeddings, i.e.,M =

∑i ei · E[i]. We then concatenate M to the

hidden representation of each token and performemotion cause tagging with a final dense layer, i.e.,ci = softmax(Wc′ [hi;M ]+bc′), where ; is the con-catenation operator and Wc′ ∈ R|C|×(dBERT+de)

and bc′ ∈ R|C| are trainable parameters. In theMultiE�C model, we again do teacher forcing anduse the gold emotion labels before doing the se-quence tagging for cause detection (i.e., e is a one-hot vector where the gold emotion label has prob-ability 1 and all other emotion labels have proba-bility 0). At inference time, the model will use thepredicted emotion distribution instead.

4.3 Adapted Knowledge Models

Recent work has shown that fine-tuning pre-trainedlanguage models such as GPT-2 on knowledgegraph tuples such as ConceptNet (Li et al.,2016) or ATOMIC (Sap et al., 2018) allows

these models to express their implicit knowledgedirectly (Bosselut et al., 2019). These adaptedknowledge models (e.g., COMET (Bosselut et al.,2019)) can produce common-sense knowledgeon-demand for any entity, relation or event.Considering that common-sense knowledge playsan important role in understanding implicitlyexpressed emotions and the reasons for thoseemotions, we explore the use of common-senseknowledge for our tasks, in particular the use ofCOMET adaptively pre-trained on the ATOMICevent-centric knowledge base. ATOMIC’s eventrelations include “xReact” and “oReact”, whichdescribe the feelings of certain entities after theinput event occurs. For example, ATOMIC’sauthors present the example of <PersonX paysPersonY a compliment, xReact, PersonX willfeel good>. xReact refers to the feelings of theprimary entity in the event, and oReact refers tothe feelings of others (in this instance, oReactyields “PersonY will feel flattered”). For example,using the headline “Sudan protests: Outrage astroops open fire on protestors", COMET-ATOMICoutputs that PersonX feels justified, PersonX feelsangry, Others feel angry, and so on (Figure 4). Touse this knowledge model in our task, we modifyour approach by reframing our single-sequenceclassification task as a sequence-pair classificationtask (for which BERT can be used directly). Wefeed our input headlines into COMET-ATOMIC(using the model weights released by the au-thors), collect the top two outputs for xReactand oReact using beam search decoding, andthen feed them into BERT alongside the inputheadlines, as a second sequence using the SEPtoken. That is, our input to BERT is now X =[[CLS], x1, x2, ..., xn, [SEP], z1, z2, ..., zm, [SEP]],where zi are the m WordPiece tokens of ourCOMET output and are preprocessed in the sameway as xi. We hypothesize that, since pre-trainedBERT is trained with a next sentence predictionobjective, expressing the COMET outputs as agrammatical sentence will help BERT make betteruse of them, so we formulate this second sequenceas complete sentences (e.g., “This person feels...Others feel...”) (Figure 4).

This approach allows us incorporate informa-tion from COMET into all our single- and multi-task BERT-based models; the example shownin Figure 4 is our MultiC�E model. Werefer to the COMET variants of these mod-

3980

Emotion Macro F1 Emotion Accuracy Cause Span F1BERT 37.25 ± 1.30 38.50 ± 0.84 37.49 ± 1.94BERTCOMET 37.74 ± 0.84 38.50 ± 1.14 39.27 ± 1.85Multi 36.91 ± 1.48 38.34 ± 1.94 38.35 ± 3.89MultiC�E 37.74 ± 2.12 38.74 ± 2.07 39.08 ± 3.73MultiE�C 38.26 ± 3.28 39.69 ± 3.41 38.83 ± 1.60MultiCOMET 37.06 ± 2.04 39.05 ± 0.98 39.50 ± 2.25MultiCOMET

C�E 39.26* ± 1.13 40.79 ± 2.17 38.68 ± 1.36MultiCOMET

E�C 37.44 ± 1.37 38.58 ± 1.44 36.27 ± 1.31

Table 1: The results of our models, averaged over five runs with the same five distinct random seeds. The modelwith the highest mean performance under each metric is bolded. Results marked with a * are statistically significantabove the single-task BERT baseline by the paired t-test (p < 0.05).

els as: BERTCOMET (single-task models) andMultiCOMET , MultiCOMET

C�E , MultiCOMETE�C for

the three multi-task models.

5 Experimental Setup

Evaluation Metrics For emotion classification,we report macro-averaged F1 and accuracy. Forcause tagging, we report exact span-level F1 (whichwe refer to as span F1), as developed for namedentity recognition (e.g., Tjong Kim Sang andDe Meulder (2003)), where a span is marked ascorrect if and only if its type and span boundariesmatch the gold exactly4.

Training and Hyperparameter Selection Theclassification layers are initialized randomly froma uniform distribution over [−0.07, 0.07]5, and allthe parameters are trained on our dataset for upto 20 epochs, with early stopping based on theperformance on the validation data (macro F1 foremotion, span F1 for cause). All models are trainedwith the Adam optimizer (Kingma and Ba, 2015).We highlight again that for our MultiC�E andMultiE�C models, we use teacher forced duringtraining to avoid cascading training error. Becausethe subset of the data we use is relatively small, wefollow current best practices for dealing with neuralmodels on small data and select hyperparametersand models using the average performance of fivemodels with different fixed random seeds on thedevelopment set. We then base our models’ per-

4Our cause tagging task has only one type, “cause”, asGoodNewsEveryone is aggregated such that each data pointhas exactly one emotion-cause pair. We note that this problemformulation leaves open the possibility of multiple emotion-cause pairs.

5The default initialization from the gluon pack-age: https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/index.html

formance on the average of the results from thesefive runs (e.g., reported emotion F1 is the averageof the emotion F1 scores for each of our five runs).For our joint models, since our novel models re-volve around using one task as input for the other,we separately tune two sets of hyperparameters foreach model, one based on each of the single-taskmetrics, yielding, for example, one Multi model op-timized for predicting emotion and one optimizedfor predicting cause. The hyperparameters we tuneare dropout in our linear layers, initial learning rateof the optimizer, COMET relation type, lambdaweight for our multi-task models, and the type ofpooler for emotion classification (enumerated insubsection 4.1).

6 Results

We present the results of our models in Table 16.We see that the overall best model for eachtask is a multi-task adapted knowledge model,with MultiCOMET

C�E performing best for emotion(which is a statistically significant improvementover BERT by the paired t-test, p < 0.05) andMultiCOMET performing best for cause. These re-sults seem to support our two hypotheses: 1) emo-tion recognition and emotion cause detection caninform each other and 2) common-sense knowledgeis helpful to infer the emotion and the cause for thatemotion expressed in text. Specifically, we noticethat MultiC�E alone does not outperform BERT oneither cause or emotion, but MultiCOMET

C�E outper-forms both BERT and MultiC�E on both tasks. Forcause, we also see additional benefits of common-

6Oberländer and Klinger (2020) report an F1 score of 34in this problem setting on this dataset, but on a larger subsetof the data (as they do not limit themselves to the Ekmanemotions) and so we cannot directly compare our work totheirs.

3981

Figure 5: Performance of the BERT and MultiCOMETC�E

models on emotion classification.

sense reasoning alone: BERTCOMET outperformsBERT (multi-task modeling alone, Multi, also out-performs BERT for this task) and MultiCOMET

outperforms Multi. These results speak to the dif-ferences between the two tasks, suggesting thatcommon-sense reasoning, which aims to generatesimplicit emotions, and cause information may becomplementary for emotion detection, but that forcause tagging, common-sense reasoning and givenemotion information may overlap. The common-sense reasoning we have used in this task (xReactand oReact from ATOMIC) is expressed as possi-ble emotional reactions to an input situation, so thismakes intuitive sense.

Finally, we also present per-emotion results forour best model for each task (MultiCOMET

C�E foremotion and MultiCOMET for cause) against thesingle-task BERT baselines in Figure 5 and Fig-ure 6; these per-emotion scores are again the av-erage performance of models trained with each ofour five random seeds. We see that each task im-proves on a different set of emotions: for emotionclassification MultiCOMET

C�E consistently improvesover BERT by a significant margin on joy and to alesser extent on anger and sadness. Meanwhile, forcause tagging, MultiCOMET improves over BERTon anger, disgust, and fear, while yielding verysimilar performance on the rest of the emotions.

7 Analysis and Discussion

In order to understand the impact of common-sensereasoning and multi-task modeling for the twotasks, we provide several types of analysis in ad-dition to our results in section 6. First, we includeexamples of our various models’ outputs showcas-ing the impact of our methods (subsection 7.1).

Figure 6: Performance of the BERT and MultiCOMET

models on cause tagging, broken down by emotion.

Second, we carry out an analysis of the dataset,focusing on the impact of label variation amongmultiple annotators on the models’ performance(subsection 7.2).

7.1 Example Outputs

We provide some example outputs from our sys-tems for both cause and emotion in Table 2; thevarious Multi models have been grouped togetherfor readability and because they often producesimilar outputs, but the outputs for every modelare available in the appendix. In the first exam-ple, the addition of COMET to BERT informs themodel enough to choose the gold emotion label;in the third and fourth, either COMET or multi-task learning is enough to help the model selectkey words that should be included in the cause (re-turn and triple shooting). We also particularly notethe second example, in which multi-task learningis needed both for the BERT and BERTCOMET

models to be able to correctly predict the gold emo-tion. This suggests that for cause, both common-sense reasoning and emotion classification maycarry overlapping useful information for cause tag-ging, while for emotion, different instances may behelped more by different aspects of our models.

7.2 Label Agreement

Upon inspection of the GoodNewsEveryone data,we discover significant variation in the emotion la-bels produced by annotators as cautioned by theauthors in their original publication7. From ourinspection of the development data, we see recur-

7While the authors selected data according to agreementon the emotion labeling task, they found that in only 75% ofcases do at least 3 annotators agree, with diminishing rates ofagreement for more annotators.

3982

BERT Multitask BERTCOMET MultitaskCOMET

Mexico reels from shooting attack in El Pasofear

negative surprise negative surprise fear fearInsane video shows Viking Sky cruise ship thrown into chaos at sea

fearnegative surprise fear negative surprise fear

Durant could return for Game 3positive surprise

for game could return for gameDan Fagan: Triple shooting near New Orleans School yet another sign of city’s crime problem

negative surpriseschool yet another sign

of city’s crime: triple shooting near new orleans school yet another sign of city’s

crime

Table 2: Example outputs from our systems. For each example, the gold cause is highlighted in yellow and thegold emotion is given under the text; the first two examples give our models’ emotion outputs; the latter two,their causes. Joined cells show that multiple models produced the same output. To make this table easier to read,“Multitask” here may refer to Multi, MultiE�C , or MultiC�E (details on selection and results for each individualmodel available in appendix; most multi-task models gave similar outputs).

Metric BERT BERTCOMET Multi MultiE�C MultiC�E MultiCOM MultiCOMETE�C MultiCOMET

C�E

Acc.(Gold) 38.50 38.50 38.34 39.68 38.74 39.05 38.58 40.79

Acc.(¬Gold) 23.48 23.24 22.37 21.11 22.85 21.26 22.45 20.08

Table 3: Comparison of gold accuracy and non-gold (¬gold) accuracy for our emotion classification models.

ring cases where different annotators give directlyopposing labels for the same input, depending onhow they interpret the headline and whose emo-tions they choose to focus on. For example, ourdevelopment set includes the following example:Simona Stuns Serena at Wimbledon: Game, Setand “Best Match” for Halep. The gold adjudi-cated emotion label for this example is negativesurprise, but annotators actually included multipleprimary and secondary emotion labels includingjoy, negative surprise, positive surprise, pride, andshame, which can be understood as various emo-tions felt by the two entities participant in the event(Simona Halep and Serena Williams). For this in-put, COMET suggests xReact may be happy orproud and oReact may be happy — these reactionsare likely most appropriate for tennis player Si-mona Halep, but not the only possible emotion thatcan be inferred from the headline.

Inspired by the variation in the data, we com-pute also models’ accuracy using the human an-notations that did not agree with the gold (i.e., apredicted emotion label is correct if it was sug-gested by a human annotator but was not part of

a majority vote to be included in the gold). Wedenote this ¬Gold, and we compare the perfor-mance of our models with respect to Gold and¬Gold. We present the results of this analysis inTable 38. In this table, a higher ¬Gold accuracymeans that the model is more likely to produceemotion labels that were not the gold but were sug-gested by some annotator. First of all, we note thatall models have a relatively high ¬Gold accuracy(about half the magnitude of their gold accuracy);we believe this reflects the wide variety of anno-tations given by the annotators. We see a trade-off between the Gold and ¬Gold accuracy, andwe note that generally the single-task models havehigher ¬Gold accuracy and the COMET-enhancedmulti-task models have higher Gold accuracy. Thissuggests that our language models have generalknowledge about emotion already, but that apply-ing common-sense knowledge helps pare down thespace of plausible outputs to those that are mostcommonly selected by human annotators. Recall

8Note that we perform this analysis on just one of our fiveruns of the model, so the accuracy numbers do not exactlycorrespond to those in Table 1.

3983

that this dataset was annotated by taking the mostfrequent of the annotator-provided emotion labels.Further, since the multi-task models have higherGold accuracy and lower ¬Gold accuracy than thesingle-task models, this suggests that also predict-ing the cause of an emotion causes the model tonarrow down the space of possible emotion labelsto only those that are most common.

8 Conclusions and Future Work

We present a common-sense knowledge-enhancedmulti-task framework for joint emotion detectionand emotion cause tagging. Our inclusion ofcommon-sense reasoning through COMET, com-bined with multi-task learning, yields performancegains on both tasks including significant gains onemotion classification. We highlight the fact thatthis work frames the cause extraction task as a spantagging task, allowing for the future possibility ofincluding multiple emotion-cause pairs per input ormultiple causes per emotion and allowing the causeto take on any grammatical role. Finally, we presentan analysis of our dataset and models, showing thatlabeling emotion and its semantic roles is a hardtask with annotator variability, but that common-sense knowledge helps language models focus onthe most prominent emotions according to humanannotators. In future work, we hope to exploreways to integrate common-sense knowledge moreinnately into our classifiers and ways to apply thesemodels to other fine-grained emotion tasks such asdetecting the experiencer or the target of an emo-tion.

Acknowledgements

We would like to thank our reviewers as well as themembers of Amazon AI team for their constructiveand insightful feedback.

Ethical Considerations

Our intended use for this work is as a tool to helpunderstand emotions expressed in text. We proposethat it may be useful for things like product reviews(where producers and consumers can rapidly assessreviews for aspects of their products to improve orexpand), disaster relief (where those in need of helpfrom any type of disaster can benefit if relief agentscan understand what events are causing negativeemotions, during and after the initial disaster), andpolicymaking (where constituents can benefit ifpolicymakers can see real data about what policies

are helpful or not and act in their interests). Theseapplications do depend on the intentions of the user,and a malicious actor may certainly misuse the abil-ity to (accurately or inaccurately) detect emotionsand their causes. We do not feel it responsible topublicly list the ways in which this may happenin this paper. We also believe that regulators andoperators of this technology should be aware that itis still in its nascent stages and does not representan infallible oracle — the predictions of this andany model should be reviewed by humans in theloop, and we feel that general public awareness ofthe limitations and mistakes of these models mayhelp mitigate any possible harm. If these modelsare inaccurate, they will output either the incorrectemotion or the incorrect cause; blindly trusting themodel’s predictions without examining them maylead to unfair consequences in any of the aboveapplications (e.g., failure to help someone whosetext is misclassified as positive surprise during anatural disaster or a worsened product or policy ifcauses are incorrectly predicted). We additionallynote that in its current form, this work is intendedto detect the emotions that are expressed in text(headlines), and not those of the reader.

We concede that the data used in this work con-sists of news headlines and may not be the mostadaptable to the use cases we describe above; wecaution that models trained on these data will likelyrequire domain adaptation to perform well in othersettings. Bostan et al. (2020) report that their datacomes from the Media Bias Chart9, which reportsthat their news sources contain a mix of politicalviews, rated by annotators who also self-reported amix of political views. We note that these data areall United States-based and in English. Bostan et al.(2020) do sub-select the news articles according toimpact on Twitter and Reddit, which have their ownuser-base biases10, typically towards young, whiteAmerican men; therefore, the data is more likely tobe relevant to these demographics. The languageused in headlines will likely most resemble Stan-dard American English as well, and therefore ourmodels will be difficult to use directly on otherdialects and vernaculars.

9https://www.adfontesmedia.com/about-the-interactive-media-bias-chart/

10https://www.pewresearch.org/internet/fact-sheet/social-media/

3984

ReferencesMuhammad Abdul-Mageed and Lyle Ungar. 2017.

EmoNet: Fine-grained emotion detection with gatedrecurrent neural networks. In Proceedings of the55th Annual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers), pages718–728, Vancouver, Canada. Association for Com-putational Linguistics.

Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat.2005. Emotions from text: Machine learning fortext-based emotion prediction. In Proceedings of theConference on Human Language Technology andEmpirical Methods in Natural Language Processing,HLT ’05, page 579–586, USA. Association for Com-putational Linguistics.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2015. Neural machine translation by jointlylearning to align and translate. In 3rd Inter-national Conference on Learning Representations,ICLR 2015, San Diego, CA, USA, May 7-9, 2015,Conference Track Proceedings.

Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai-tanya Malaviya, Asli Çelikyilmaz, and Yejin Choi.2019. COMET: commonsense transformers forautomatic knowledge graph construction. In Pro-ceedings of the 57th Conference of the Associationfor Computational Linguistics, ACL 2019, Florence,Italy, July 28- August 2, 2019, Volume 1: Long Pa-pers, pages 4762–4779. Association for Computa-tional Linguistics.

Laura Ana Maria Bostan, Evgeny Kim, and RomanKlinger. 2020. Goodnewseveryone: A corpus ofnews headlines annotated with emotions, semanticroles, and reader perception. In Proceedings of The12th Language Resources and Evaluation Confer-ence, LREC 2020, Marseille, France, May 11-16,2020, pages 1554–1566. European Language Re-sources Association.

Ying Chen, Wenjun Hou, Xiyao Cheng, and ShoushanLi. 2018. Joint learning for emotion classificationand emotion cause detection. In Proceedings of the2018 Conference on Empirical Methods in NaturalLanguage Processing, pages 646–651, Brussels, Bel-gium. Association for Computational Linguistics.

Dorottya Demszky, Dana Movshovitz-Attias, Jeong-woo Ko, Alan Cowen, Gaurav Nemade, and SujithRavi. 2020. GoEmotions: A dataset of fine-grainedemotions. In Proceedings of the 58th Annual Meet-ing of the Association for Computational Linguistics,pages 4040–4054, Online. Association for Computa-tional Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human Language

Technologies, NAACL-HLT 2019, Minneapolis, MN,USA, June 2-7, 2019, Volume 1 (Long and Short Pa-pers), pages 4171–4186. Association for Computa-tional Linguistics.

Zixiang Ding, Rui Xia, and Jianfei Yu. 2020. ECPE-2D: Emotion-cause pair extraction based on jointtwo-dimensional representation, interaction and pre-diction. In Proceedings of the 58th Annual Meet-ing of the Association for Computational Linguistics,pages 3161–3170, Online. Association for Computa-tional Linguistics.

Chuang Fan, Chaofa Yuan, Jiachen Du, Lin Gui, MinYang, and Ruifeng Xu. 2020. Transition-based di-rected graph construction for emotion-cause pair ex-traction. In Proceedings of the 58th Annual Meet-ing of the Association for Computational Linguistics,pages 3707–3717, Online. Association for Computa-tional Linguistics.

D. Ghazi, Diana Inkpen, and S. Szpakowicz. 2015.Detecting emotion stimuli in emotion-bearing sen-tences. In CICLing.

Deepanway Ghosal, Navonil Majumder, AlexanderGelbukh, Rada Mihalcea, and Soujanya Poria. 2020.COSMIC: COmmonSense knowledge for eMotionidentification in conversations. In Findings of theAssociation for Computational Linguistics: EMNLP2020, pages 2470–2481, Online. Association forComputational Linguistics.

Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, andYu Zhou. 2016. Event-driven emotion cause extrac-tion with corpus construction. In Proceedings ofthe 2016 Conference on Empirical Methods in Natu-ral Language Processing, pages 1639–1649, Austin,Texas. Association for Computational Linguistics.

Evgeny Kim and Roman Klinger. 2018. Who feelswhat and why? annotation of a literature corpuswith semantic roles of emotions. In Proceedingsof the 27th International Conference on Computa-tional Linguistics, pages 1345–1359, Santa Fe, NewMexico, USA. Association for Computational Lin-guistics.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In 3rd Inter-national Conference on Learning Representations,ICLR 2015, San Diego, CA, USA, May 7-9, 2015,Conference Track Proceedings.

Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mo-hammad. 2014. Sentiment analysis of short infor-mal texts. J. Artif. Int. Res., 50(1):723–762.

Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel.2016. Commonsense knowledge base completion.In Proceedings of the 54th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 1445–1455, Berlin, Germany.Association for Computational Linguistics.

3985

Saif Mohammad and Peter D. Turney. 2013. Crowd-sourcing a word-emotion association lexicon. CoRR,abs/1308.6297.

Saif Mohammad, Xiaodan Zhu, and Joel Martin. 2014.Semantic role labeling of emotions in tweets. In Pro-ceedings of the 5th Workshop on Computational Ap-proaches to Subjectivity, Sentiment and Social Me-dia Analysis, pages 32–41.

Saif M. Mohammad, Felipe Bravo-Marquez, Mo-hammad Salameh, and Svetlana Kiritchenko. 2018.Semeval-2018 Task 1: Affect in tweets. In Proceed-ings of International Workshop on Semantic Evalua-tion (SemEval-2018), New Orleans, LA, USA.

Laura Ana Maria Oberländer and Roman Klinger. 2020.Token sequence labeling vs. clause classification forEnglish emotion stimulus detection. In Proceedingsof the Ninth Joint Conference on Lexical and Com-putational Semantics, pages 58–70.

Laura Ana Maria Oberländer, Kevin Reich, and Ro-man Klinger. 2020. Experiencers, stimuli, or tar-gets: Which semantic roles enable machine learningto infer the emotions? In Proceedings of the ThirdWorkshop on Computational Modeling of People’sOpinions, Personality, and Emotion’s in Social Me-dia, pages 119–128.

Soujanya Poria, Devamanyu Hazarika, Navonil Ma-jumder, and Rada Mihalcea. 2020a. Beneath thetip of the iceberg: Current challenges and new di-rections in sentiment analysis research.

Soujanya Poria, Navonil Majumder, Devamanyu Haz-arika, Deepanway Ghosal, Rishabh Bhardwaj, Sam-son Yu Bai Jian, Romila Ghosh, Niyati Chhaya,Alexander Gelbukh, and Rada Mihalcea. 2020b.Recognizing emotion cause in conversations.

Lance A. Ramshaw and Mitchell P. Marcus. 1995.Text chunking using transformation-based learning.CoRR, cmp-lg/9505040.

Sara Rosenthal, Saif M. Mohammad, Preslav Nakov,Alan Ritter, Svetlana Kiritchenko, and Veselin Stoy-anov. 2019. Semeval-2015 task 10: Sentiment anal-ysis in twitter. CoRR, abs/1912.02387.

Irene Russo, Tommaso Caselli, Francesco Rubino, Es-ter Boldrini, and Patricio Martínez-Barco. 2011.EMOCause: An easy-adaptable approach to extractemotion cause contexts. In Proceedings of the 2ndWorkshop on Computational Approaches to Subjec-tivity and Sentiment Analysis (WASSA 2.011), pages153–160, Portland, Oregon. Association for Compu-tational Linguistics.

Maarten Sap, Ronan LeBras, Emily Allaway, Chan-dra Bhagavatula, Nicholas Lourie, Hannah Rashkin,Brendan Roof, Noah A. Smith, and Yejin Choi. 2018.ATOMIC: an atlas of machine commonsense for if-then reasoning. CoRR, abs/1811.00146.

Carlo Strapparava and Rada Mihalcea. 2010. Annotat-ing and identifying emotions in text. In Giuliano Ar-mano, Marco de Gemmis, Giovanni Semeraro, andEloisa Vargiu, editors, Intelligent Information Ac-cess, volume 301, pages 21–38.

Erik F. Tjong Kim Sang and Fien De Meulder.2003. Introduction to the CoNLL-2003 shared task:Language-independent named entity recognition. InProceedings of the Seventh Conference on Natu-ral Language Learning at HLT-NAACL 2003, pages142–147.

Penghui Wei, Jiahao Zhao, and Wenji Mao. 2020.Effective inter-clause modeling for end-to-endemotion-cause pair extraction. In Proceedings of the58th Annual Meeting of the Association for Compu-tational Linguistics, pages 3171–3181, Online. As-sociation for Computational Linguistics.

Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen,Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,Teven Le Scao, Sylvain Gugger, Mariama Drame,Quentin Lhoest, and Alexander Rush. 2020. Trans-formers: State-of-the-art natural language process-ing. In Proceedings of the 2020 Conference on Em-pirical Methods in Natural Language Processing:System Demonstrations, pages 38–45, Online. Asso-ciation for Computational Linguistics.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V.Le, Mohammad Norouzi, Wolfgang Macherey,Maxim Krikun, Yuan Cao, Qin Gao, KlausMacherey, Jeff Klingner, Apurva Shah, Melvin John-son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws,Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, KeithStevens, George Kurian, Nishant Patil, Wei Wang,Cliff Young, Jason Smith, Jason Riesa, Alex Rud-nick, Oriol Vinyals, Greg Corrado, Macduff Hughes,and Jeffrey Dean. 2016. Google’s neural machinetranslation system: Bridging the gap between humanand machine translation. CoRR, abs/1609.08144.

Rui Xia and Zixiang Ding. 2019. Emotion-cause pairextraction: A new task to emotion analysis in texts.CoRR, abs/1906.01267.

Rui Xia, Mengran Zhang, and Zixiang Ding. 2019.RTHN: A rnn-transformer hierarchical network foremotion cause extraction. CoRR, abs/1906.01236.

Peng Xu, Andrea Madotto, Chien-Sheng Wu, Ji HoPark, and Pascale Fung. 2018. Emo2vec: Learn-ing generalized emotion representation by multi-tasktraining. CoRR, abs/1809.04505.

Chaofa Yuan, Chuang Fan, Jianzhu Bao, and RuifengXu. 2020. Emotion-cause pair extraction as se-quence labeling based on a novel tagging scheme.In Proceedings of the 2020 Conference on EmpiricalMethods in Natural Language Processing (EMNLP),pages 3568–3573, Online. Association for Computa-tional Linguistics.

3986

A Appendix

A.1 Hyperparameter TuningWe include descriptions of our hyperparameter tun-ing setup and the selected hyperparmeters for eachof our models in Table 4; we note that single-taskcause models (BERTC and COMETC) do not tunethe pooler, all single-task models do not tunethe lambda parameter, and all non-common-sensemodels do not tune comet_relations. The pa-rameters selected by all of our models can be seenin Table 5, Table 6, and Table 7. All of our modelsare trained with minibatches of size b = 32.

We used Bayesian optimization as implementedby Amazon SageMaker11 to tune these parameters,giving the learning rate a logarithmic scale and thedropout and lambda a linear one and allowing 75iterations of parameter choice before selecting thesetting with the best performance on the develop-ment set. Each individual instance of each modelconsisted of five different restarts with five distinctrandom seeds; one of these instances took approxi-mately five minutes on a single Tesla V100 GPU,for a total of about 6.25 GPU-hours per model andthus 87.5 GPU-hours overall (since each multi-taskmodel was trained twice: once optimized for emo-tion and once optimized for cause).

A.2 Model SizesOur models’ sizes are dominated by BERT-base,which has 110 million trainable parameters (Devlinet al., 2019). We note that our trainable dense layersthat interface with BERT have sizes 768 × 7 foremotion classification, 768 × 3 for cause tagging,and 1068 × 7 for our MultiE�C models, while ouremotion embedding matrixE has 300× 7 trainableparameters. Our fine-tuning process does continueto tune all of BERT’s parameters.

A.3 Extended ExamplesWe include the output of all models for our fourselected examples in subsection 7.1 in Table 8, Ta-ble 9, Table 10, and Table 11.

11https://aws.amazon.com/sagemaker/

3987

Parameter Name Type Range or Valuespooler Categorical [cls, mean, max, attention]learning rate Continuous [10−6, 10−4]dropout Continuous [0, 0.9]lambda Continuous [0.1, 0.9]comet_relations Categorical [xReact, oReact, both]

Table 4: Our hyperparameter search ranges.

Model Target Task Parameter Name Parameter Value

BERTE Emotionpooler clsdropout 0.8999992513311351lr 2.0872134970009262× 10−5

BERTC Causedropout 0.04011659404129298lr 9.609926650689472× 10−5

BERTCOMETE Emotion

pooler clsdropout 0.6467089448672897lr 3.548213539029209× 10−5

comet_relations both

BERTCOMETC Cause

dropout 0.8806119007595122lr 9.913585728926367× 10−5

comet_relations xReact

Table 5: The selected hyperparameter values for our single-task models.

Model Target Task Parameter Name Parameter Value

Multi

Emotion

pooler meandropout 0.1438975482079587lr 2.170218150294524× 10−5

lambda 0.3736515054477897

Cause

pooler clsdropout 0.8929935089177194lr 9.929740332732521× 10−5

lambda 0.6103686494768474

MultiE�C

Emotion

pooler maxdropout 0.2511612834815036lr 3.179072019077849× 10−5

lambda 0.4938386162506444

Cause

pooler maxdropout 0.763419047616446lr 8.680439371509037× 10−5

lambda 0.1341940851689314

MultiC�E

Emotion

pooler maxdropout 0.8138762283528274lr 4.2586257586160994× 10−5

lambda 0.8531247637209994

Cause

pooler meandropout 0.6992099059226856lr 9.859155309987275× 10−5

lambda 0.4855821360212248

Table 6: The selected hyperparameter values for our multi-task BERT models.

3988

Model Target Task Parameter Name Parameter Value

MultiCOMET

Emotion

pooler maxdropout 0.22350077887111716lr 3.137385699389837× 10−5

lambda 0.7676911585403968comet_relations both

Cause

pooler meandropout 0.8891347000216091lr 8.123006047625093× 10−5

lambda 0.1comet_relations both

MultiCOMETE�C

Emotion

pooler meandropout 0.1372637910712323lr 3.0408118480380588× 10−5

lambda 0.8968243966922735comet_relations both

Cause

pooler maxdropout 0.5319636087561394lr 7.581334242472624× 10−5

lambda 0.10896064677810494comet_relations both

MultiCOMETC�E

Emotion

pooler clsdropout 0.7359624181177503lr 1.9853909769532754× 10−5

lambda 0.7947522633173147comet_relations both

Cause

pooler maxdropout 0.01896406469706125lr 8.360862387915605× 10−5

lambda 0.14588492191321054comet_relations oReact

Table 7: The selected hyperparameter values for our multi-task COMET models.

Mexico reels from shooting attack in El Pasofear

Model OutputBERT negative surpriseBERTCOMET fearMulti negative surpriseMultiC�E negative surpriseMultiE�C negative surpriseMultiCOMET fearMultiCOMET

C�E fearMultiCOMET

E�C fear

Table 8: Full model outputs for our first provided example.

3989

Insane video shows Viking Sky cruise ship thrown into chaos at seafear

Model OutputBERT negative surpriseBERTCOMET negative surpriseMulti fearMultiC�E negative surpriseMultiE�C fearMultiCOMET fearMultiCOMET

C�E fearMultiCOMET

E�C negative surprise

Table 9: Full model outputs for our second provided example.

Durant could return for Game 3positive surprise

Model OutputBERT for gameBERTCOMET could return for gameMulti could return for gameMultiC�E could return for gameMultiE�C could return for gameMultiCOMET could return for gameMultiCOMET

C�E could return for gameMultiCOMET

E�C could return for game

Table 10: Full model outputs for our third provided example.

Dan Fagan: Triple shooting near New Orleans School yet another sign of city’s crime problemnegative surprise

Model OutputBERT school yet another sign of city’s crimeBERTCOMET : triple shooting near new orleans school yet another sign of city’s crimeMulti shooting near new orleans school yet another sign of city’s crimeMultiC�E : triple shooting near new orleans school yet another sign of city’s crimeMultiE�C : triple shooting near new orleans school yet another sign of city’s crimeMultiCOMET : triple shooting near new orleans school yet another sign of city’s crimeMultiCOMET

C�E : triple shooting near new orleans school yet another sign of city’s crimeMultiCOMET

E�C : triple shooting near new orleans school yet another sign of city’s crime

Table 11: Full model outputs for our fourth provided example.