On Irony Detection in Social Media

Paolo Rosso Natural Language Engineering Lab – PRHLT Research Center

Technical University of Valencia http://www.dsic.upv.es/~prosso/

Artificial Intelligence & Natural Language (AINL)

Moscow, 12th September 2014

Outline

• Figurative language: humour, irony,…

• Irony: linguistic device for polarity negation

• Verbal vs. situational irony

• Irony in social media

• Benchmark activities and projects on irony detection

• Recent works on irony detection: 2013 & 2014

Figurative language processing

• Figurative vs. natural language: figurative vs. literal meaning

• Humour, irony, metaphor etc.

• No facial expression or voice pitch

• Irony and opinion mining: implicit negation of polarity in sentiment analysis

• Opposition (lack of an explicit negation marker), incongruity, intentionality, ambiguity, unexpectedness, etc.

• Verbal vs. situational irony: e.g. A vegetarian having a heart attack outside Mc Donald’s / Burger King…

Visual / situational irony (incongruity)

Intentionality (most of the times…)

Picture taken at Kurskiy railway station in Moscow, one month ago

Verbal + visual irony (unexpectedness)

Irony in Russian

Irony in Russian

Irony in Russian (incongruity)

Irony and reputation in social media

Toyota's new slogan; moving forward (even if u don't want to); hahahaha :)

'Toyota; moving forward.' Yeah because you have faulty brakes and jammed accelerators. :P

My car broke down! Nooooooooooo! I bought a Toyota so that it wouldn't brake down.:(

CERN recruiting engineers from Toyota for further improvements to their particle accelerator :P IamconCERNed

#Toyota tweets

Irony and hashtags (the wisdom of crowds)

Irony and virality: viral effect / viral marketing

Irony, sarcasm or satire If you find it hard to laugh at yourself,

I would be happy to do it for you

My mother never saw the irony

in calling me a son-of-a-bitch

Humour and irony: one-liners

Jesus saves, and at today's prices, that's a miracle!

Love is blind, but marriage is a real eye-opener.

Drugs may lead to nowhere, but at least it's a scenic route.

Become a computer programmer and never see the world again.

My software never has bugs; it just develops random features.

Sex is one of the nine reasons for reincarnation; the other eight are unimportant.

I've got the body of a god ...unfortunately is Buddha.

Humour and irony: one-liners: some pattern

Jesus saves, and at today's prices, that's a miracle! [ambiguity]

Love is blind, but marriage is a real eye-opener. [antonymy]

Drugs may lead to nowhere, but at least it's a scenic route. [human weakness]

Become a computer programmer and never see the world again. [common topic / community]

My software never has bugs; it just develops random features. [??]

Sex is one of the nine reasons for reincarnation; the other eight are unimportant. [language]

I've got the body of a god ...unfortunately is Buddha. [irony]

Humour and irony: more examples

I’m on a thirty day diet. So far, I have lost 15 days

Change is inevitable, except from a vending machine

Children in the back seats of cars cause accidents, but accidents in the back seats of cars cause children.

Don’t worry about what people think. They don’t do it very often.

I feel so miserable without you, it’s almost like having you here.

Sometimes I need what only you can provide: your absence.

Humour and irony: more patterns

I’m on a thirty day diet. So far, I have lost 15 days.

Change is inevitable, except from a vending machine.

Children in the back seats of cars cause accidents, but accidents in the back seats of cars cause children.

Don’t worry about what people think. They don’t do it very often.

I feel so miserable without you, it’s almost like having you here.

Sometimes I need what only you can provide: your absence.

Irony and humour: more patterns

I’m on a thirty day diet. So far, I have lost 15 days. [incongruity]

Change is inevitable, except from a vending machine. [ambiguity]

Children in the back seats of cars cause accidents, but accidents in the back seats of cars cause children. [syntactic ambiguity]

Don’t worry about what people think. They don’t do it very often. [irony]

I feel so miserable without you, it’s almost like having you here. [irony]

Sometimes I need what only you can provide: your absence. [irony]

State-of-the-art

Humour recognition & generation:

Phonological, incongruity, semantics (Binsted, Mihalcea, Strapparava).

Irony, sarcasm, satire detection:

Similes, onomatopoeic expressions, headlines (Veale, Hao, Carvalho, Tsur)

Irony and humour: some features

N-grams: frequent sequences of words

Descriptors: tuned up sequences of words

POS n-grams: POS templates

Polarity: polarity of words

Affectiveness: emotional content (WordNet Affect)

Pleasantness: degree of pleasure (Whissel’s dictionary)

Funniness: relationship between humor and irony (humour domains and lexical ambiguity)

Tested on Amazon viral effect corpus: (Reyes and Rosso, 2013)

Irony detection: more ambitious features • Signatures: Pointedness (typographical marks: punctuation or emoticons); Counter-

factuality (discursive marks: adverbs implying negation: nevertheless); Temporal compression: opposition in time (adverbs of time: suddenly, now).

• Unexpectedness: Temporal imbalance (opposition in a same document); Contextual imbalance (inconsistencies within a context – semantic relatedness).

• Style: Character n-grams (c-grams); Skip n-grams (s-grams); Polarity s-grams (ps-sgrams).

• Emotional contexts: Activation (degree of response that humans have under an emotional state); Imagery (how difficult is to form a mental picture of a given word); Pleasantness (degree of pleasure produced by words).

Examples • Activation:

My male(1.55) ego(2.00) so eager(2.25) to let(1.70) it be stated(2.00) that I’m THE MAN(1.8750) but won’t allow(1.00) my pride(1.90) to admit(1.66) that being egotistical(0) is a weakness(1.75) ...

• Imagery:

Yesterday(1.6) was the official(1.4) first(1.6) day(2.6) of spring(2.8)... and there was over a foot(2.8) of snow(3.0) on the ground(2.4).

• Pleasantness :

The guy(1.9000) who(1.8889) called(2.0000) me Ricky(0) Martin(0) has(1.7778) a blind(1.0000) lunch(2.1667) date(2.33).

Results (Twitter)

Tested on Twitter corpus (Reyes et al., 2013)

Some references Reyes A., Rosso P., Buscaldi D. (2012). From Humor Recognition to Irony Detection:

The Figurative Language of Social Media. In: Data & Knowledge Engineering, 74:1-12 Reyes A., Rosso P. (2013). Making Objective Decisions from Subjective Data: Detecting

Irony in Customers Reviews. In: Journal on Decision Support Systems, 53(4):754–760 Reyes A., Rosso P., Veale T. (2013). A Multidimensional Approach for Detecting Irony

in Twitter. In: Language Resources and Evaluation, 47(1):239-268 Reyes A., Rosso P. (2014). On the Difficulty of Automatically Detecting Irony: Beyond a

Simple Case of Negation. In: Knowledge and Information Systems, 40(3): 595-614

http://www. dsic.upv.es/~prosso/

Benchmark activities on irony detection • Pilot task @ Sentipolc: Evalita 2014

http://www.evalita.it/2014/tasks/sentipolc

Organisers: Viviana Patti (Università di Torino), Andrea Bolioli (CELI),

Malvina Nissim (Università di Bologna), Valerio Basile (University of Groningen),

Paolo Rosso (Universitat Politècnica de València)

• Sentiment Analysis of Figurative Language in Twitter: Task 11 @ SemEval 2015

http://alt.qcri.org/semeval2015/task11

Organisers: Tony Veale (University College Dublin), John Barnden (University of Birmingham), Antonio Reyes (ISIT), Ekaterina Shutova (UC Berkeley),

Paolo Rosso (Universitat Politècnica de València)

Projects on irony/sarcasm detection (in US)

Army Research Office (ARO)

Sociolinguistically Informed Natural Language Processing:

Automating Irony Detection

http://www.reddit.com/r/irony

Secret Service seeks Twitter sarcasm detector

http://www.bbc.com/news/technology-27711109

http://www.washingtonpost.com/blogs/the-fix/wp/2014/06/03/the-secret-service-wants-software-that-detects-social-media-sarcasm-yeah-sure-it-will-work/

The tweet should be detected as ironic…

J. M. Whalen, P. M. Pexman, A. J. Gill & S. Nowson

Behavior & Information Technology (32)6: 560-569, 2013.

Verbal irony use in personal blogs

71 regular bloggers (24 male and 47 female) from North America, UK, Australia and New Zeeland.

The utterance was only counted as ironic if it was clear that a literal interpretation was not intended.

Hyperbole was the ironic form most frequently used by bloggers (for instance wrt sarcasm)

Inter-annotator agreement for identifying that an utterance was ironic: 89.57% (on the 25% of the blogs, selected randomly)

Inter-annotator agreement on the category: 98.36%

#Irony or #Sarcasm A quantitative and qualitative study based on Twitter

Po-Ya Angela Wang

Proc. 27th Pacific Asia Conference on Language, Information, and

Computation (PACLIC 27), 2013

Irony & Sarcasm

Identify similarities and distinctions

Quantitative Sentiment Analysis

Qualitative content analysis

Special way of language creativity

Interaction between cognition and language

Speaker intention plays an important role

Irony is an umbrella term that covers Sarcasm

Corpus: 500 tweets #irony & 500 tweets #sarcasm

Tagging: crowdsourcing (participants are asked to judge how good the example is to be ironic/sarcastic).

They used a lexicon of 2600 positive words and 4783 negative words: difference between positive and negative words in a tweet is the sentiment score of the tweet.

Interest to understand how speakers use sentiment words in these types of language creativity.

Sarcastic tweets use more positive words but ironic tweets use more neutral

The positive words used in tweets seems to represent the aggressive intention

Sarcasm as contrast between a positive sentiment and a negative situation

E. Riloff, A. Qadir, P. Surve, L. De Silva, N. Gilbert & R. Huang

Proc. Conference on Empirical Methods in Natural Language

Processing (EMNLP), 2013

Sarcastic tweets often express a positive sentiment in reference to a negative situation

The goal is to identify sarcasm that arises from the contrast between a positive sentiment referring to a negative situation

Identify stereotypically negative “situations” (unenjoyable or undesirable)

#sarcasm reveals the intended sarcasm, but we do not always have the benefit of an explicit sarcasm label

Positive sentiment word with a negative activity or state

Oh how I love being ignored #sarcasm

Absolutely adore it when my bus is late #sarcasm

Authors Focus on positive sentiments that are expressed as a verb phrase or as a predicative expression and negative activities or states that can be complement to a verb phrase.

Assume sarcasm probably arises from positive/negative contrast and exploit syntactic structure to extract phrases that are likely to have contrasting polarity

Harvest the n-grams that follow the word “love” as negative situation candidates, then selected the best of them using a scoring metric and add them to a list of negative situation phrases.

Collected 1,600 tweets with a sarcasm hashtag (#sarcasm or #sarcastic) and 1,600 without this hashtags.

Created a gold standard data set of manually annotated tweets (sarcasm hashtags were removed)

They perform a set of experiments, one of which consist in label a tweet as sarcastic if contains a positive sentiment phrase in close proximity to a negative situation phrase, both extracted from their bootstrapping algorithm. Achieves a precision of 70%

Contrasting a positive sentiment with a negative situation seems to be a key element of sarcasm.

Modelling irony in Twitter

F. Barbieri & H. Saggion

Proc. of the Student Research Workshop at the 14th conference of the European Chapter of the Association for Computational Linguistics (EACL), 2014.

Irony

Model uses seven groups of features to represent each tweet:

*Frequency: gap between rare and common words

*Written-spoken: written-spoken style uses

*Intensity: intensity of adverbs and adjectives

Structure: length, punctuation, emoticons

Sentiments: gap between positive and negative terms

Synonyms: common vs. rare synonyms use

Ambiguity: measure of possible ambiguity

Dataset used: (Reyes et al., 2013)

Decision tree

Education Humor Irony Politics

* not used before for irony detection

Frequency

ANC: American National Corpus Frequency Data

to measure the frequency of word usage Written-Spoken

Intensity Intensity of Potts1 adjectives and adverbs scale based on star ratings on service and product reviews

1 http://www.stanford.edu/~cgpotts/data/wordnetscales/

Synonyms WordNet & ANC

WordNet

Structure

Ambiguity

Sentiments SentiWordNet

Model

Education Humour Politics

P R F1 P R F1 P R F1

Reyes et al. 0.76 0.66 0.70 0.78 0.74 0.76 0.75 0.71 0.73

Authors 0.73 0.73 0.73 0.75 0.75 0.75 0.75 0.75 0.75

Modelling sarcasm in Twitter, a novel approach

F. Barbieri & H. Saggion

Proc. of the 5th Computational Approaches to Subjectivity, Sentiment & Social Media, WASSA 2014.

Experiments

Sarcasm vs

Education Humor Irony Newspaper Politics

The best results are obtained when distinguished Sarcasm from Newspaper tweets (F1: 0.97)

Difficulty in distinguishing sarcastic tweets from ironic ones (F1 : 0.62)

Relevant features to detect sarcasm against irony are two:

Use of adverbs: sarcasm uses less adverbs but more intense

Sentiment scores: sarcastic tweet are denoted by more positive sentiments than irony

An impact analysis of features in a classification approach to irony detection in product reviews

K. Buschmeier, P. Cimiano & R. Klinger Proc. of the 5th Workshop on Computational Approaches to

Subjectivity, Sentiment and Social Media Analysis Association for Computational Linguistics, WASSA 2014

Aim: to contribute to a deeper understanding of the linguistic properties of irony and sarcasm as linguistic phenomena and their corpus based evaluation and verification.

Authors analyze the impact of a number of features which have been proposed in previous research on irony detection

Automatic classification of a product review corpus from Amazon, by Filatova (Irony and sarcasm: Corpus generation and analysis using crowdsourcing, LREC 2012)

Irony detection as a supervised classification problem

Features

Imbalance between the overall polarity of words in the review and the star-rating

Hyperbole indicates the occurrence of a sequence of three positive or negative words in a row

Quotes indicates that up to two consecutive adjectives or nouns in quotation marks have a positive or negative polarity

Pos/Neg & Punctuation span of up to four words contains at least one positive(negative) but no negative (positive) word and ends with at least two exclamation marks

Pos/Neg & Ellipsis indicates that such a positive or negative span ends with an ellipsis (“…”)

Emoticon indicates the occurrence of an emoticon

Punctuation conveys the presence of an ellipsis as well as multiple question or exclamation marks or a combination of the latter two

Interjection indicates the occurrence of terms like “wow” and “huh”

Laughter measures onomatopoeia as well as acronyms of grin or laughter

Bag of words

Classifiers: SMV, Naïve Bayes, Logistic Regression, Decision Tree and Random Forest Classifier

Corpus: 1254 Amazon Reviews, 437 ironic utterances.

Baselines: Star-rating relies only on the number of stars assigned in the review as feature. Bag-of-words exploits only the unigrams in the text as features, sentiment word count, All (all features)

Performed experiments using different feature set combinations for the different classifiers.

The best result is achieved by using the star-rating together with bag-of-words and all features with a logistic regression approach (F1: 0.74)

L. Alba-Juez & S. Attardo

Evaluation in Context (Chapter 5)

John Benjamins Publishing Company, 2014

The evaluative palette of verbal irony

Irony

Negative

Most frequent and common type of verbal irony

Typical examples of sarcasm where an apparently positive comment expresses a negative criticism or judgment of a person, a thing or a situation.

Positive Positive evaluation of a given person, thing or situation.

Frequently found in family discourse

Neutral

No intention of criticizing or praising any participant, thing, or situation

The utterance may include some kind or overt evaluation (very distant from either a positive or a critical negative position).

Irony

Negative

After Peter betrays his friend Tom, Tom says to Peter: You’re certainly my best friend ever!

Tom is using negative irony in order to express his very negative evaluation of the way in which Peter has behaved.

Positive

Daniel comes back home from school and shows his father his report-card, which is full of As, to which his father reacts in the following manner: Father: Daniel, I’m really worried; your grades are terrible! (with blank face) Daniel: (giggles) Thank you, Dad The father is trying to express his pride for his son’s success, an ironic act that is clearly understood by Daniel, as can be deduced from hi answer and reaction.

Neutral

From Blaise Pascal Letter XVI. The letter is longer than usual because

I didn’t have the time to make it shorter

Seems to be not intention of criticizing or praising any participant, thing, or situation. Pascal was using fine irony in order to show wittiness, and therefore be funny.

Purpose: to see if the native speakers of each of the 2 languages distinguished between the ironic and non-ironic utterances, as well as to verify whether there was any significant difference in the identification of irony’s polarity.

Designed a questionnaire (both in English and Spanish) based on 20 situations. Ten of these situations contained some ironic utterances that could be related to a positive, a negative or a neutral evaluative stance, and the other ten were used as distractors.

38 native speakers of English and 56 of Spanish

Participant would have to classify each of the 20 situations according to the labels ironic/sarcastic, polite/impolite, aggressive/not aggressive, humorous/non-humorous

Conclude that speakers can identify reliably ironical and non ironical utterances

Results reveal that there seems to be no difference between the identification of negative irony and that positive and/or neutral irony, which not only supports authors’ hypothesis in favor of the existence of different “evaluative values” in ironic speech acts, but in fact supports a much stronger claim namely that positive and neutral irony are not significantly harder to identify than negative irony

Getting reliable annotations for sarcasm in online dialogues

R. Swanson, S. Lukin, L. Eisenberg, T. Chase Corcoran & M. A. Walker

Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC) 2014

Report the first study of the issues involved with achieving high reliability labels for sarcasm in online dialogue

Authors used Internet Argument Corpus (IAC), a large corpus of online social and political dialogues. The initial IAC annotation involved 10,003 Quote-Response (Q-R) pairs where Mechanical Turkers were shown seven Q-R pairs and asked to judge whether the response was sarcastic or not.

Turkers were not given additional definitions of the meaning of sarcasm

A subset of 25 new annotations was used to compare the different reliability measures on gold standard data in terms of accuracy as a function of the number of Turker annotations.

Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis

Diana Maynard and Mark A. Greenwood

Proceedings of the Ninth International Conference on

Language Resources and Evaluation, LREC 2014

Consider in particular the effect of sentiment and sarcasm contained in hashtags, and have developed a hashtag tokenizer for GATE, so that sentiment and sarcasm found within hashtags can be detected more easily

Tweets labeled with the hashtag #irony typically do not refer to verbal irony, but to situational irony. Collected a corpus of 257 tweets containing the hashtag #irony, and found that only 2 tweets contained clear instances of verbal irony, about 25% involved clear situational irony, while about 75% referred to extra-contextual information, so that the meaning was not clear.

Sarcasm Detection on Czech and English Twitter

Tomáš Ptácek, Ivan Habernal and Jun Hong

Proc. 25th Int. Conf. on Computational Linguistics

COLING-2014

Chinese Irony Corpus Construction and Ironic Structure Analysis

Y. Tang and H. Chen

Proc. 25th Int. Conf. on Computational Linguistics

COLING-2014

Emotions and Irony per Gender in Facebook

F. Rangel, I. Hernández, P. Rosso & A. Reyes

Proc. Workshop on Emotion, Social Signals, Sentiment

& Linked Open Data (ES³LOD), LREC-2014

Emotions & irony per gender in FB Anger

Fear

Disgust

Surprise

Joy

Sadness

+

+

Ekman 6 basic emotions + no-emotion

Statistics wrt irony

ironic/non-ironic comments (2/3 annotators)

ironic comments per topic and gender (2/3 annotators) ironic comments per emotion (2/3 annotators)

ironic comments per annotator

Inter-annotator agreement: irony ‣ Fleiss Kappa: It allows multiple annotators (three in our case) and binary variables

(ironic / non-ironic)

‣ We obtained a value of 0.0989 -> very low index of agreement

‣ Irony is quite subjective and depends on annotators, their moods, linguistic and cultural context: we did not provide a common definition for irony

‣ Contextual information was not provided, only individual comments

‣ Males tended to be more ironic than females (in this corpus)

‣ The category politics is the one with more negative emotions and irony

(in Spain? Difficult to believe it… #irony)

‣ EmIroGeFB Facebook corpus tagged with Emotions, Irony and Gender:

Inter-annotator agreement: irony & emotional comments

‣ Kappa Diaz-Sidorov (it allows to calculate concordance for more than two annotators, in our case three, with multiple not mutually exclusive categories, the six basic emotions, in the subset of comments identified as ironic

‣ We obtained a negative value of -0.0660: there is no agreement among

annotators

Spasibo! Questions?

Irony (and its detection) is fun! Enjoy it! Enjoy task 11 @ SemEval-2015 http://alt.qcri.org/semeval2015/task11/

Paolo Rosso: prosso@dsic.upv.es

Artificial Intelligence & Natural Language (AINL) Moscow, 12th September 2014