detecting sarcasm in multimodal social platforms

Eureka Presentation

ROSSANO SCHIFANELLA, Paloma de Juan, Joel Tetreault, Liangliang Cao

@ACM Multimedia 2016, Amsterdam

DETECTING SARCASM IN MULTIMODAL SOCIAL PLATFORMS

Eureka Presentation

SARCASM

Eureka Presentation

Because who doesn’t love finishing the slides late at night the day before the talk… #acmmm2016 #hangover

Eureka Presentation

WHAT IS SARCASM?

LITERAL INTENDED≠

Eureka Presentation

Great day today

Eureka Presentation

LEXICAL and LINGUISTIC MARKERS

Eureka Presentation

INTERJECTIONS, INTENSIFIERS, HYPERBOLES

Well, really great day today

Eureka Presentation

Great day today !?!?!?!?

PUNCTUATION

Eureka Presentation

CONTEXT

Eureka Presentation

Great day today! #epicfail

HASHTAGS

Eureka Presentation

Great day today! #winning

HASHTAGS

Eureka Presentation

Great day today! 😭 ⛈ ☔

EMOJIS

Eureka Presentation

Great day today! 😍 🏖☀

EMOJIS

Eureka Presentation

Great day today

Third car accident in a mile!

1

2

PREVIOUS POSTS

@RSCHIFAN

@RSCHIFAN

Eureka Presentation

AUTHOR PROFILE, PROPENSITY TO SARCASTIC UTTERANCES

Great day today

Well this is not stressful at all #sarcasm

1

3

@RSCHIFAN

@RSCHIFAN

2 I looooove Trump’s hair! #sarcasm@RSCHIFAN

Eureka Presentation

SOCIAL MEDIA IS MULTIMODAL

Eureka Presentation

METADATA VISUALS

TEXT

Eureka Presentation

Great day today

Eureka Presentation

Great day today

Eureka Presentation

Text+Image

Image as a contextual clue

Eureka Presentation

POSTS CONTAINING #SARCASM OR #SARCASTIC

DATA

517K 63K 20K99% 40% 7.56%

TEXT+IMAGE TEXT+IMAGE TEXT+IMAGE

Eureka Presentation

CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

1

Eureka Presentation

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YE

SN

O

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YES

NO

TEXT-ONLY

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION

It was a beautiful spring day today! I almost went out outside in shorts it was so nice! 😭 😣 😭 #spring #sarnia #winter #allgonein24hours

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YES

NO

TEXT+IMAGE

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION

Seriously cute cat just wandered into my garden, sweet little thing 😍 #cat #photogenic #cute #garden

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION

So happy I brought the nice weather back with me...

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key -Not Sarcastic: #sarcasm is not always

sufficient to mark the content as sarcastic, users have often their own definition of sarcasm that is close to humour, fun, silly content.

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YE

SNO NOT SARCASTIC

Eureka Presentation

COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement

CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

1

2

Eureka Presentation

ASK THE CROWD!

1K POSTS

5 JUDGEMENTS

Eureka Presentation

SECOND EXPERIMENT

For all the posts that are judged not sarcastic in the previous step, show the text and the image

FIRST EXPERIMENT

Show only the textual component of a post

Eureka Presentation

Text+Image 37,4%

Text Only 37,8%

Not Sarcastic 24,8%

Text+Image 44,5%

Text Only 23,6%

Not Sarcastic 31,9%

\

Eureka Presentation

COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement

DETECT SARCASMSVM Fusion+Deep learning fusion approaches

CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

1

2

3

HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?

1 SVM

Eureka Presentation

-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION

NLP FEATURES VISUAL SEMANTIC FEATURES

-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL

NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE

Eureka Presentation

-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION

NLP FEATURES VISUAL SEMANTIC FEATURES

-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL

NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE

+

LINEAR SVM

+FEATURES VECTOR

FUSION

HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?

2 DEEP

LEARNING

Eureka Presentation

1 K. Chateld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: delving deep into convolutional nets. In BMVC, 2014.

Adapted Visual Representation1 (trained on ImageNet)

NLP Multilayer Perceptron (based on unigrams)

CO

NC

ATE

NA

TIO

N

LAY

ER

NO

N-L

INE

AR

LA

YE

RS

SARCASM DETECTION

Eureka Presentation

EVALUATIONGOLD SET

2K EXAMPLES

Eureka Presentation

D-50 D-80 D-100

baseline (1,2-grams) 81.7 82.5 80.2

baseline + VSF +6% +6.3% +4.3%

D-50 D-80 D-100

baseline (1,2-grams) 88.8 86.0 84.4

baseline + VSF -0,04% +2.1% +6.2%

SVM

1

Eureka Presentation

D-50 D-80 D-100

baseline 77 74.6 74.8

baseline + AVR +1% +5.1% +3.7%

D-50 D-80 D-100

baseline 75.8 74.6 75.5

baseline + AVR +2.4% +1.4% -1%

DEEP LEARNING

2

Eureka Presentation

Future Directions

SVM: IMPROVE THE FUSION METHOD, ADD SEMANTICS

DEEP LEARNING APPROACH: A LOT TO DO!

VISUAL SENTIMENT CONCEPTS

AUTOMATIC GENERATION OF SARCASTIC IMAGE CAPTION

SARCASTIC CONVERSATIONAL BOTS

Eureka Presentation

Questions?

@rschifan

http://www.di.unito.it/~schifane

schifane@di.unito.it

THANKS FOR THE VERY INTERESTING TALK!