detecting sarcasm in multimodal social platforms

of 44/44
Eureka Presentation ROSSANO SCHIFANELLA, Paloma de Juan, Joel Tetreault, Liangliang Cao @ACM Multimedia 2016, Amsterdam DETECTING SARCASM IN MULTIMODAL SOCIAL PLATFORMS

Post on 16-Feb-2017

144 views

Embed Size (px)

TRANSCRIPT

  • Eureka Presentation

    ROSSANO SCHIFANELLA, Paloma de Juan, Joel Tetreault, Liangliang Cao

    @ACM Multimedia 2016, Amsterdam

    DETECTING SARCASM IN MULTIMODAL SOCIAL PLATFORMS

  • Eureka Presentation

    SARCASM

  • Eureka Presentation

    Because who doesnt love finishing the slides late at night the day before the talk #acmmm2016 #hangover

  • Eureka Presentation

    WHAT IS SARCASM?

    LITERAL INTENDED

  • Eureka Presentation

    Great day today

  • Eureka Presentation

    LEXICAL and LINGUISTIC MARKERS

  • Eureka Presentation

    INTERJECTIONS, INTENSIFIERS, HYPERBOLES

    Well, really great day today

  • Eureka Presentation

    Great day today !?!?!?!?

    PUNCTUATION

  • Eureka Presentation

    CONTEXT

  • Eureka Presentation

    Great day today! #epicfail

    HASHTAGS

  • Eureka Presentation

    Great day today! #winning

    HASHTAGS

  • Eureka Presentation

    Great day today!

    EMOJIS

  • Eureka Presentation

    Great day today!

    EMOJIS

  • Eureka Presentation

    Great day today

    Third car accident in a mile!

    1

    2

    PREVIOUS POSTS

    @RSCHIFAN

    @RSCHIFAN

  • Eureka Presentation

    AUTHOR PROFILE, PROPENSITY TO SARCASTIC UTTERANCES

    Great day today

    Well this is not stressful at all #sarcasm

    1

    3

    @RSCHIFAN

    @RSCHIFAN

    2 I looooove Trumps hair! #[email protected]

  • Eureka Presentation

    SOCIAL MEDIA IS MULTIMODAL

  • Eureka Presentation

    METADATA VISUALS

    TEXT

  • Eureka Presentation

    Great day today

  • Eureka Presentation

    Great day today

  • Eureka Presentation

    Text+Image

    Image as a contextual clue

  • Eureka Presentation

    POSTS CONTAINING #SARCASM OR #SARCASTIC

    DATA

    517K 63K 20K99% 40% 7.56%

    TEXT+IMAGE TEXT+IMAGE TEXT+IMAGE

  • Eureka Presentation

    CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

    1

  • Eureka Presentation

    -100 posts per platform -Two questions:

    A. Is the text enough? B. Does the image help?

    MANUAL ANNOTATION IS THE TEXT ENOUGH?

    YES NO

    DO

    ES T

    HE IM

    AG

    E H

    ELP

    ?

    YE

    SN

    O

  • Eureka Presentation

    -Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

    TAKEAWAYS

    -100 posts per platform -Two questions:

    A. Is the text enough? B. Does the image help?

    MANUAL ANNOTATION IS THE TEXT ENOUGH?

    YES NO

    DO

    ES T

    HE IM

    AG

    E H

    ELP

    ?

    YES

    NO

    TEXT-ONLY

  • Eureka Presentation

    -Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

    TAKEAWAYS

    -100 posts per platform -Two questions:

    A. Is the text enough? B. Does the image help?

    MANUAL ANNOTATION

    It was a beautiful spring day today! I almost went out outside in shorts it was so nice! #spring #sarnia #winter #allgonein24hours

  • Eureka Presentation

    -Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

    -Text+Image: multimodality is key

    TAKEAWAYS

    -100 posts per platform -Two questions:

    A. Is the text enough? B. Does the image help?

    MANUAL ANNOTATION IS THE TEXT ENOUGH?

    YES NO

    DO

    ES T

    HE IM

    AG

    E H

    ELP

    ?

    YES

    NO

    TEXT+IMAGE

  • Eureka Presentation

    -Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

    -Text+Image: multimodality is key

    TAKEAWAYS

    -100 posts per platform -Two questions:

    A. Is the text enough? B. Does the image help?

    MANUAL ANNOTATION

    Seriously cute cat just wandered into my garden, sweet little thing #cat #photogenic #cute #garden

  • Eureka Presentation

    -Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

    -Text+Image: multimodality is key

    TAKEAWAYS

    -100 posts per platform -Two questions:

    A. Is the text enough? B. Does the image help?

    MANUAL ANNOTATION

    So happy I brought the nice weather back with me...

  • Eureka Presentation

    -Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

    -Text+Image: multimodality is key -Not Sarcastic: #sarcasm is not always

    sufficient to mark the content as sarcastic, users have often their own definition of sarcasm that is close to humour, fun, silly content.

    TAKEAWAYS

    -100 posts per platform -Two questions:

    A. Is the text enough? B. Does the image help?

    MANUAL ANNOTATION IS THE TEXT ENOUGH?

    YES NO

    DO

    ES T

    HE IM

    AG

    E H

    ELP

    ?

    YE

    SNO NOT SARCASTIC

  • Eureka Presentation

    COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement

    CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

    1

    2

  • Eureka Presentation

    ASK THE CROWD!

    1K POSTS

    5 JUDGEMENTS

  • Eureka Presentation

    SECOND EXPERIMENT

    For all the posts that are judged not sarcastic in the previous step, show the text and the image

    FIRST EXPERIMENT

    Show only the textual component of a post

  • Eureka Presentation

    Text+Image 37,4%

    Text Only 37,8%

    Not Sarcastic 24,8%

    Text+Image 44,5%

    Text Only 23,6%

    Not Sarcastic 31,9%

    \

  • Eureka Presentation

    COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement

    DETECT SARCASMSVM Fusion+Deep learning fusion approaches

    CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

    1

    2

    3

  • HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?

    1 SVM

  • Eureka Presentation

    -LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION

    NLP FEATURES VISUAL SEMANTIC FEATURES

    -YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL

    NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE

  • Eureka Presentation

    -LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION

    NLP FEATURES VISUAL SEMANTIC FEATURES

    -YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL

    NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE

    +

    LINEAR SVM

    +FEATURES VECTOR

    FUSION

  • HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?

    2 DEEP

    LEARNING

  • Eureka Presentation

    1 K. Chateld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: delving deep into convolutional nets. In BMVC, 2014.

    Adapted Visual Representation1 (trained on ImageNet)

    NLP Multilayer Perceptron (based on unigrams)

    CO

    NC

    ATE

    NA

    TIO

    N

    LAY

    ER

    NO

    N-L

    INE

    AR

    LA

    YE

    RS

    SARCASM DETECTION

  • Eureka Presentation

    EVALUATIONGOLD SET

    2K EXAMPLES

  • Eureka Presentation

    D-50 D-80 D-100

    baseline (1,2-grams) 81.7 82.5 80.2

    baseline + VSF +6% +6.3% +4.3%

    D-50 D-80 D-100

    baseline (1,2-grams) 88.8 86.0 84.4

    baseline + VSF -0,04% +2.1% +6.2%

    SVM

    1

  • Eureka Presentation

    D-50 D-80 D-100

    baseline 77 74.6 74.8

    baseline + AVR +1% +5.1% +3.7%

    D-50 D-80 D-100

    baseline 75.8 74.6 75.5

    baseline + AVR +2.4% +1.4% -1%

    DEEP LEARNING

    2

  • Eureka Presentation

    Future Directions

    SVM: IMPROVE THE FUSION METHOD, ADD SEMANTICS

    DEEP LEARNING APPROACH: A LOT TO DO!

    VISUAL SENTIMENT CONCEPTS

    AUTOMATIC GENERATION OF SARCASTIC IMAGE CAPTION

    SARCASTIC CONVERSATIONAL BOTS

  • Eureka Presentation

    Questions?

    @rschifan

    http://www.di.unito.it/~schifane

    [email protected]

    THANKS FOR THE VERY INTERESTING TALK!